Voice processing methods, terminal devices and storage media

By obtaining user characteristic information to determine the target voice style, the problem that voice playback cannot adapt to the user's state in the existing technology is solved, and rich voice styles and high-quality voice playback are achieved.

CN112489619BActive Publication Date: 2026-11-10SHANGHAI TRANSSION CO LTD
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202011334384.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-11-24
Publication Date
2026-11-10
Estimated Expiration
2040-11-24

AI Technical Summary

Technical Problem

Existing speech processing and synthesis technologies cannot automatically switch to an appropriate speech style based on the user's state, resulting in speech playback lacking rich speech styles and emotions, and thus having insufficient quality.

Method used

By acquiring the user's preset characteristic information, including daily routine information, contextual information, emotional characteristics, personality traits, gender, and age, the target speech style for voice broadcasting is determined, and a speech document with the target speech style is generated through speech synthesis technology.

Benefits of technology

It enables automatic switching of the appropriate voice style based on the user's current state, enriching the style and emotion of voice playback and improving the quality of voice playback.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112489619B_ABST
    Figure CN112489619B_ABST
Patent Text Reader

Abstract

This application discloses a voice processing method, a terminal device, and a storage medium. The voice processing method includes: S11, acquiring preset feature information of the user; S12, determining the target voice style for voice playback based on the preset feature information. This application can not only automatically switch to an appropriate voice style according to the user's current state, but also provide a rich variety of voice styles, which is beneficial to improving the quality of voice playback.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of speech processing and synthesis technology, and in particular to a speech processing method, as well as a terminal device and a readable storage medium based on the speech processing method. Background Technology

[0002] In recent years, with the continuous development of the online audio-visual market, speech processing and speech synthesis technologies have been widely applied in people's daily lives, such as online children's books, online novels, online storytelling, and online news. However, current speech playback based on speech processing and synthesis technologies is mostly mechanical, emotionless, flat, and monotonous reading of text content. It is limited to converting text into comprehensible speech, lacks rich speech styles, and cannot automatically switch to an appropriate speech style based on the user's current state, thus failing to further improve the quality of speech playback.

[0003] The preceding description is intended to provide general background information and does not necessarily constitute prior art. Summary of the Invention

[0004] In view of this, this application provides a voice processing method, terminal device and storage medium to solve the problem of not being able to select an appropriate voice style according to the user's state to achieve voice playback.

[0005] This application provides a speech processing method, including:

[0006] S11. Obtain the user's preset feature information;

[0007] S12. Determine the target voice style for voice broadcasting based on the user's preset feature information.

[0008] Optionally, the preset feature information includes at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age.

[0009] Optionally, step S11 includes at least one of the following:

[0010] Obtain preset feature information based on the selected operation and / or input operation;

[0011] Preset feature information is obtained based on historical habits and / or sensors;

[0012] Acquire the user's voice data and obtain preset feature information based on the voice data.

[0013] Optionally, before step S12, the method further includes: selecting a suitable document to be played according to the target speech style, identifying and extracting the text content in the document to be played, and / or performing speech synthesis on the text content to generate a speech document with the target speech style.

[0014] Optionally, the method further includes: identifying and extracting the text content of the resource to be played;

[0015] After step S12, the following steps are performed: speech synthesis on the text content to generate a speech document with the target speech style.

[0016] Optionally, after step S12, the method further includes:

[0017] Select the appropriate document to be played based on preset feature information;

[0018] Identify and extract text content from the document to be played; and / or, perform speech synthesis on the text content to generate a speech document with the target speech style.

[0019] Optionally, before step S12, the method further includes: obtaining preset feature information of the document to be played;

[0020] Determine whether the preset feature information of the document to be played conflicts with the user's preset feature information;

[0021] If not, then proceed to step S12; and / or,

[0022] If so, the default strategy will be executed.

[0023] Optionally, obtaining the preset feature information of the document to be played includes at least one of the following:

[0024] Obtain its preset feature information based on the preset category tags of the document to be played;

[0025] Determine the preset feature information of the document to be played based on its text content.

[0026] Optionally, the preset strategy includes at least one of the following:

[0027] Perform step S12;

[0028] The target speech style is determined based on the selected instructions;

[0029] The target speech style is determined based on the preset feature information of the document to be played;

[0030] Use the default voice style as the target voice style when playing voice messages.

[0031] This application provides a speech processing method, including:

[0032] S21. Obtain the preset feature information of the document to be played;

[0033] S22. Determine the target speech style for voice playback based on the preset feature information of the document to be played.

[0034] Optionally, step S21 includes at least one of the following:

[0035] Obtain its preset feature information based on the preset category tags of the document to be played;

[0036] Determine the preset feature information of the document to be played based on its text content.

[0037] Optionally, the preset feature information includes at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age.

[0038] Optionally, before step S22, the method further includes: obtaining the user's preset feature information;

[0039] Determine whether the document to be played conflicts with the user's preset feature information;

[0040] If not, then proceed to step S22; and / or,

[0041] If so, the default strategy will be executed.

[0042] Optionally, the preset strategy includes at least one of the following:

[0043] Perform step S22;

[0044] The target speech style is determined based on the selected instructions;

[0045] Determine the target speech style based on the user's preset feature information;

[0046] Use the default voice style as the target voice style when playing voice messages.

[0047] This application provides a terminal device including a memory and a processor. The memory stores a voice processing program, which, when executed by the processor, implements the steps of any of the above-mentioned voice processing methods.

[0048] This application provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described speech processing methods.

[0049] As described above, the speech processing method, terminal device, and storage medium of this application determine the target speech style for speech playback based on the user's preset feature information. The preset feature information identifies the user's current state. This not only enables automatic switching of the appropriate speech style according to the user's current state, but also provides a rich variety of speech styles, which is beneficial to improving the quality of speech playback. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 A schematic diagram of the hardware structure of a terminal device for implementing various embodiments of this application;

[0052] Figure 2 A communication network system architecture diagram provided in this application embodiment;

[0053] Figure 3 This is a schematic flowchart of the speech processing method according to the first embodiment of this application;

[0054] Figure 4 This is a schematic diagram of the operation interface of an embodiment of obtaining preset feature information of a user in this application;

[0055] Figure 5 This is a schematic diagram of the operation interface of another embodiment of the present application for obtaining user preset feature information;

[0056] Figure 6 This is a schematic diagram of the operation interface of another embodiment of the present application for obtaining the user's preset feature information;

[0057] Figure 7 This is a schematic diagram of the user interface of an embodiment of the present application for determining a target speech style;

[0058] Figure 8 This is a schematic flowchart of the speech processing method according to the second embodiment of this application;

[0059] Figure 9 This is a schematic flowchart of the speech processing method according to the third embodiment of this application;

[0060] Figure 10 This is a schematic flowchart of the speech processing method according to the fourth embodiment of this application;

[0061] Figure 11 This is a schematic flowchart of the speech processing method according to the fifth embodiment of this application;

[0062] Figure 12 This is a schematic flowchart of the speech processing method according to the sixth embodiment of this application.

[0063] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0064] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0065] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0066] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0067] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0068] It should be noted that step designations such as S11 and S12 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S12 first and then S11, etc., but these should all be within the protection scope of this application.

[0069] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0070] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0071] Terminal devices can be implemented in various forms. For example, the terminal devices described in this application may include mobile terminals such as mobile phones, tablets, laptops, handheld computers, personal digital assistants (PDAs), portable media players (PMPs), navigation devices, wearable devices, smart bracelets, pedometers, etc., as well as fixed terminals such as digital TVs and desktop computers.

[0072] The following description will use a mobile terminal as an example. Those skilled in the art will understand that, apart from elements specifically designed for mobile purposes, the construction according to the embodiments of this application can also be applied to fixed-type terminals.

[0073] Please see Figure 1This is a schematic diagram of the hardware structure of a mobile terminal implementing various embodiments of this application. The mobile terminal 100 may include: an RF (Radio Frequency) unit 101, a WiFi module 102, an audio output unit 103, an A / V (Audio / Video) input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, a processor 110, and a power supply 111, etc. Those skilled in the art will understand that... Figure 1 The mobile terminal structure shown does not constitute a limitation on the mobile terminal. The mobile terminal may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0074] The following is combined Figure 1 A detailed introduction to each component of the mobile terminal:

[0075] The radio frequency unit 101 can be used for receiving and transmitting signals during information transmission or calls. Specifically, it receives downlink information from the base station and processes it with the processor 110; additionally, it transmits uplink data to the base station. Typically, the radio frequency unit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier, and a duplexer. Furthermore, the radio frequency unit 101 can also communicate wirelessly with networks and other devices. The aforementioned wireless communications may use any communication standard or protocol, including but not limited to GSM (Global System of Mobile communication), GPRS (General Packet Radio Service), CDMA2000 (Code Division Multiple Access 2000), WCDMA (Wideband Code Division Multiple Access), TD-SCDMA (Time Division-Synchronous Code Division Multiple Access), FDD-LTE (Frequency Division Duplexing-Long Term Evolution), and TDD-LTE (Time Division Duplexing-Long Term Evolution).

[0076] WiFi is a short-range wireless transmission technology. Mobile terminals, through the WiFi module 102, can help users send and receive emails, browse web pages, and access streaming media, providing users with wireless broadband internet access. Although Figure 1 WiFi module 102 is shown, but it is understood that it is not a necessary component of a mobile terminal and can be omitted as needed without changing the nature of the invention.

[0077] The audio output unit 103 can convert audio data received by the radio frequency unit 101 or the WiFi module 102 or stored in the memory 109 into audio signals and output them as sound when the mobile terminal 100 is in call signal receiving mode, call mode, recording mode, voice recognition mode, broadcast receiving mode, etc. Furthermore, the audio output unit 103 can also provide audio output related to specific functions performed by the mobile terminal 100 (e.g., call signal receiving sound, message receiving sound, etc.). The audio output unit 103 may include a speaker, a buzzer, etc.

[0078] The A / V input unit 104 is used to receive audio or video signals. The A / V input unit 104 may include a graphics processing unit (GPU) 1041 and a microphone 1042. The GPU 1041 processes image data of still images or videos acquired by an image capture device (such as a camera) in video capture mode or image capture mode. The processed image frames can be displayed on the display unit 106. The image frames processed by the GPU 1041 can be stored in the memory 109 (or other storage medium) or transmitted via the radio frequency unit 101 or the WiFi module 102. The microphone 1042 can receive sound (audio data) in operating modes such as telephone call mode, recording mode, and voice recognition mode, and can process such sound into audio data. The processed audio (voice) data can be converted into a format that can be transmitted to a mobile communication base station via the radio frequency unit 101 in telephone call mode. The microphone 1042 can implement various types of noise cancellation (or suppression) algorithms to eliminate (or suppress) noise or interference generated during the reception and transmission of audio signals.

[0079] The mobile terminal 100 also includes at least one sensor 105, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor includes an ambient light sensor and a proximity sensor. Optionally, the ambient light sensor can adjust the brightness of the display panel 1061 according to the ambient light level, and the proximity sensor can turn off the display panel 1061 and / or backlight when the mobile terminal 100 is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in various directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that recognize the phone's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc. Other sensors that may be configured in the phone, such as fingerprint sensors, pressure sensors, iris sensors, molecular sensors, gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, will not be described in detail here.

[0080] The display unit 106 is used to display information input by the user or information provided to the user. The display unit 106 may include a display panel 1061, which may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0081] User input unit 107 can be used to receive input numerical or character information, and generate key signal inputs related to user settings and function control of the mobile terminal. Specifically, user input unit 107 may include touch panel 1071 and other input devices 1072. Touch panel 1071, also known as touch screen, can collect touch operations on or near the user (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near touch panel 1071), and drive corresponding connection devices according to a pre-set program. Touch panel 1071 may include two parts: touch detection device and touch controller. Optionally, touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to touch controller; touch controller receives touch information from touch detection device, converts it into touch point coordinates, sends it to processor 110, and can receive and execute commands from processor 110. In addition, touch panel 1071 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1071, the user input unit 107 may also include other input devices 1072. Specifically, other input devices 1072 may include, but are not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc., without being limited here.

[0082] Furthermore, the touch panel 1071 may cover the display panel 1061. When the touch panel 1071 detects a touch operation on or near it, it transmits the information to the processor 110 to determine the type of touch event. Subsequently, the processor 110 provides corresponding visual output on the display panel 1061 based on the type of touch event. Although in Figure 1 In this embodiment, the touch panel 1071 and the display panel 1061 are two independent components to realize the input and output functions of the mobile terminal. However, in some embodiments, the touch panel 1071 and the display panel 1061 can be integrated to realize the input and output functions of the mobile terminal. The specific implementation is not limited here.

[0083] Interface unit 108 serves as an interface through which at least one external device can connect to mobile terminal 100. For example, the external device may include a wired or wireless headset port, an external power supply (or battery charger) port, a wired or wireless data port, a memory card port, a port for connecting a device with an identification module, an audio input / output (I / O) port, a video I / O port, a headphone port, and so on. Interface unit 108 may be used to receive input (e.g., data, power, etc.) from the external device and transmit the received input to one or more elements within mobile terminal 100, or it may be used to transmit data between mobile terminal 100 and the external device.

[0084] The memory 109 can be used to store software programs and various data. The memory 109 may primarily include a program storage area and a data storage area. Optionally, the program storage area may store the operating system, applications required for at least one function (such as sound playback, image playback, etc.), etc.; the data storage area may store data created based on the use of the mobile phone (such as audio data, phonebook, etc.). Furthermore, the memory 109 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0085] The processor 110 is the control center of the mobile terminal. It connects various parts of the mobile terminal via various interfaces and lines. By running or executing software programs and / or modules stored in the memory 109, and by calling data stored in the memory 109, it performs various functions and processes data of the mobile terminal, thereby providing overall monitoring of the mobile terminal. The processor 110 may include one or more processing units; preferably, the processor 110 may integrate an application processor and a modem processor. Optionally, the application processor mainly handles the operating system, user interface, and applications, while the modem processor mainly handles wireless communication. It is understood that the modem processor may not be integrated into the processor 110.

[0086] The mobile terminal 100 may also include a power supply 111 (such as a battery) that supplies power to various components. Preferably, the power supply 111 can be logically connected to the processor 110 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system.

[0087] although Figure 1 As not shown, the mobile terminal 100 may also include a Bluetooth module, etc., which will not be described in detail here.

[0088] To facilitate understanding of the embodiments of this application, the communication network system on which the mobile terminal of this application is based is described below.

[0089] Please see Figure 2 , Figure 2 This application provides a communication network system architecture diagram. The communication network system is an LTE system based on the universal mobile communication technology. The LTE system includes a UE (User Equipment) 201, an E-UTRAN (Evolved UMTS Terrestrial Radio Access Network) 202, an EPC (Evolved Packet Core) 203, and the operator's IP services 204, which are connected in sequence.

[0090] Specifically, UE201 can be the aforementioned terminal 100, which will not be elaborated here.

[0091] E-UTRAN202 includes eNodeB2021 and other eNodeB2022, etc. Optionally, eNodeB2021 can connect to other eNodeB2022 via backhaul (e.g., X2 interface), and eNodeB2021 connects to EPC203, providing access from UE201 to EPC203.

[0092] EPC203 may include MME (Mobility Management Entity) 2031, HSS (Home Subscriber Server) 2032, other MMEs 2033, SGW (Serving Gateway) 2034, PGW (Packet Data Network Gateway) 2035, and PCRF (Policy and Charging Rules Function) 2036, etc. Optionally, MME2031 is the control node that handles signaling between UE201 and EPC203, providing bearer and connection management. HSS2032 is used to provide registers to manage functions such as the Home Location Register (not shown in the figure) and stores user-specific information such as service characteristics and data rates. All user data can be sent through SGW2034. PGW2035 can provide UE 201 IP address allocation and other functions. PCRF2036 is the policy and charging control decision point for service data flow and IP bearer resources. It selects and provides available policy and charging control decisions for the policy and charging enforcement function unit (not shown in the figure).

[0093] IP services 204 may include the Internet, intranet, IMS (IP Multimedia Subsystem), or other IP services.

[0094] Although the above description uses the LTE system as an example, those skilled in the art should understand that this application is not only applicable to the LTE system, but also to other wireless communication systems, such as GSM, CDMA2000, WCDMA, TD-SCDMA, and future new network systems, etc., which are not limited here.

[0095] Based on the above-described mobile terminal hardware structure and communication network system, various embodiments of this application are proposed.

[0096] Please see Figure 3 This is a schematic flowchart of a voice processing method according to an embodiment of this application. The execution subject of this method can be the aforementioned terminal device 100, and it may include the following steps S11 and S12.

[0097] S11. Obtain the user's preset feature information.

[0098] Preset feature information can be considered as information that can be used to indicate the user's status. The types of preset feature information can be set by the user according to actual usage, or they can be set by default by the terminal device. The reference dimensions for user settings and terminal device default settings include, but are not limited to, at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age.

[0099] Daily routine information indicates the time a user performs behaviors such as sleeping, exercising, eating, and working. Contextual information indicates the user's current behavior and the environmental conditions in which it occurs, such as indoors, outdoors, weather, running, or walking. Emotional characteristics indicate the user's current mood, such as excitement, frustration, or anger. Personality characteristics indicate the user's personality, including but not limited to active, stable, humorous, cute, and serious types. Since people's voice styles differ significantly at different ages, age groups can be set to include four stages: infancy, youth, middle age, and old age.

[0100] The methods for obtaining these preset feature information include, but are not limited to, at least one of the following:

[0101] The first method involves acquiring preset feature information based on selection and / or input operations. In other words, the terminal device acquires preset feature information based on the user's manual operations.

[0102] In one embodiment, the terminal device may provide an operating interface; please refer to [link to relevant documentation]. Figure 4 As shown, the user interface displays several dimensions, such as contextual information, personality traits, and age, as shown in the figure.

[0103] For scenarios with a large number of dimension types that cannot all be displayed in a single frame, the terminal device can display them through multiple user interfaces. Each interface displays one dimension type and its specific options. Please refer to [link / reference]. Figure 5 As shown, daily routine information, contextual information, emotional characteristics, and personality characteristics, along with their respective specific options, are displayed on separate user interfaces. After the user completes the selection for each dimension, the terminal device switches to the user interface for the next dimension and its specific options.

[0104] Users can obtain their preset feature information by clicking to select specific options for some or all of the dimension types. For example, in Figure 5 In a specific scenario, if a user selects exercise as the activity information, outdoor as the context information, excitement as the emotional characteristic, and active as the personality characteristic, the terminal device will record and obtain the preset characteristic information as: exercise, outdoor, excitement, and active.

[0105] For another implementation method, please refer to [link / reference]. Figure 6The terminal device provides an interface that can display custom input options. After the user clicks on the custom input option, the terminal device switches to another interface that displays an input box, thereby obtaining preset feature information based on the user's input.

[0106] Preferably, the input box can display text descriptions for various dimensions, such as... Figure 6 The "contextual information," "emotional characteristics," "personality characteristics," "gender," and "age" displayed are used to prompt the user.

[0107] The method of user input is not limited in this application embodiment. For example, it can be touch keyboard input (including but not limited to Pinyin input, stroke input), or voice input.

[0108] Voice input facilitates operation for visually impaired users and others. Furthermore, before inputting each dimension category, the terminal device can verbally play a text description of that dimension category. For example, when switching to... Figure 6 When the input box interface is displayed, the terminal device plays a sound containing "Please enter the current context information", then collects the user's voice command and performs voice recognition on the information.

[0109] Of course, voice playback and voice recognition methods can also be applied to the aforementioned selection operations, for example, when switching to... Figure 4 or Figure 5 When the user interface is displayed, the terminal device plays a sound that says "Please select the current context information", then plays the specific content included in that dimension category, then collects the user's voice command and recognizes the specific content selected by the user.

[0110] It should be understood that terminal devices can also combine the aforementioned selection and input operations to obtain the user's preset feature information; for specific implementation methods, please refer to [link / reference needed]. Figures 4 to 6 As described.

[0111] The second method involves acquiring preset feature information based on historical habits and / or sensors.

[0112] Based on users' historical activity data (historical habits), AI technology is used to obtain users' activity information. For example, the time periods during which users most frequently perform behaviors such as sleeping, exercising, eating, and working within a 24-hour period each day.

[0113] Sensors include, but are not limited to, at least one of the following: temperature sensor, gravity sensor, camera, 3D facial recognition sensor, microphone, etc. Terminal devices can use one or more sensors to collect corresponding preset characteristic information of the user. For example, a gravity sensor and camera can be used to identify the user's environment and thus obtain contextual information; a camera can be used to identify the user's gender and age; a 3D facial recognition sensor can be used to collect the user's facial features and determine emotional characteristics; and a microphone can be used to collect the user's voice data and thereby identify emotional characteristics, personality traits, gender, and age.

[0114] Terminal devices can combine historical habits and sensor data to acquire preset user characteristic information. When the identification results of these two methods conflict, the result of one method can be taken as the standard, for example, preferably the sensor's identification result. A conflict means that the identification results for the same dimension are completely different. For example, if historical habits indicate that the user is currently sleeping, while the sensor identifies that the user is moving, these two methods result in conflicting identification results for the dimension of "life and sleep information." In this case, the terminal device preferably determines that the user is moving.

[0115] When the results from these two identification methods do not conflict, they can be combined. For example, if historical habits indicate that the user is eating dinner, and the sensor identifies a young woman eating indoors and feeling depressed, the final user's preset characteristic information can be: a young woman eating dinner indoors and feeling depressed. This can more accurately reflect the user's current state.

[0116] The third method involves acquiring the user's voice data and then using that data to obtain preset feature information.

[0117] Terminal devices acquire user voice data in ways including but not limited to: collecting data through sensors such as microphones, receiving data from other devices, or downloading data from the cloud.

[0118] For example, the terminal device engages in human-computer dialogue with the user or collects the user's voice when issuing voice commands, and obtains the user's voice data accordingly. The voice data is then input into a voice style prediction model to predict and judge the user's current state and output the user's preset feature information.

[0119] It should be understood that the terminal device may provide any two or all of the above three methods. When the identification results of the same dimension type conflict, the identification result of one method shall prevail, for example, the identification result of the sensor shall prevail. When the identification results of the two methods do not conflict, the identification results can be combined to more accurately reflect the user's current status.

[0120] In addition, the above three acquisition methods are only illustrative examples. Other embodiments of this application may set acquisition methods that are different from the above methods and / or have more than three types.

[0121] S12. Determine the target voice style for voice broadcasting based on the user's preset feature information.

[0122] Voice style is manifested through voice elements; in other words, different voice styles have different voice elements (at least one of loudness, pitch, and timbre). The terminal device can adjust at least one of loudness, pitch, and timbre to obtain the target voice style. For ease of description and explanation, embodiments of this application can classify voice styles as including at least one of the following: steady voice style, lively voice style, humorous voice style, cute voice style, and passionate voice style.

[0123] In a specific scenario, the terminal device can automatically inform the user of the target speech style obtained based on preset feature information. Please refer to [link / reference]. Figure 7 The terminal device will pop up a dialog box, which displays the currently determined voice style, such as "steady voice style". In order to facilitate the terminal for visually impaired people to make voice broadcasts, the user can reply by voice whether to use the currently determined voice style as the target voice style, or touch and click the "Yes" option to use it as the target voice style, or click the "No" option to not use it as the target voice style, and re-determine the voice style according to the preset feature information.

[0124] Preset feature information represents the user's current state, and each user state corresponds to a different voice style. In a given user state, the terminal device can switch to execute an appropriate voice style based on a pre-set correspondence. A rich variety of voice styles helps improve the quality of voice playback.

[0125] For example, in scenarios where facial features are used to learn a user's emotional characteristics, the terminal device can automatically switch to a personalized tone and timbre for reading aloud. When the user is in a good mood, it can automatically switch to a light and cheerful tone and timbre to read travel-related books with emotion; when the user is in a bad mood, it can automatically switch to a light and soothing tone and timbre to read lighthearted or humorous books with emotion.

[0126] For example, when the device detects that a user is running outdoors, it can automatically switch to a rousing tone and timbre to play audiobooks that the user is interested in; when it detects that a user is walking on the beach, it can automatically switch to a relaxed tone and timbre to play audiobooks that the user is interested in.

[0127] For example, when the device detects that a user has fallen asleep, it can automatically ask the user if they want to play an audiobook. If the device receives a response indicating that they do, it will play a soothing audiobook in a calming tone and emotional manner that the user is interested in. Preferably, the device can determine the user's daily screen time based on recorded sleep patterns. If the screen time exceeds the healthy limit, the device will automatically ask the user if they want to play an audiobook to protect their eyesight.

[0128] For voice playback scenarios, please refer to Figure 8 As shown, in Figure 3 Based on the described method, the speech processing method of this application may further include steps S13 and S14.

[0129] S13. Identify and extract the text content of the resource to be played.

[0130] S14. Perform speech synthesis on the text content to generate a speech document with the target speech style.

[0131] The resources to be played include at least one of the following: books, images, web pages, and URLs. In general, the resources to be played include two categories: character documents and image documents. For image documents, the characters can be identified and converted into character documents according to their display order within the image.

[0132] The text content of the resource to be played can be a character document, an audio document, or a combination of both.

[0133] The text content of the resource to be played is converted into a speech document (called the initial speech document) using speech processing technology, and then a speech document with the target speech style (called the output speech document) is generated using speech synthesis technology. This results in a rich variety of speech playback styles and high playback quality.

[0134] Taking reading books by visually impaired individuals as an example, visually impaired individuals, reading a vast amount of books in their own way, benefit from a variety of voice styles that help them understand the world more emotionally and comprehensively, enhancing their self-awareness. For instance, for technical and professional books, the target voice style could be a steady and composed one; for entertainment books, a lively and engaging one; for humorous books, a witty and humorous one; and for children's books, a cute and charming one.

[0135] Figure 9 This is a schematic flowchart illustrating the speech processing method according to the third embodiment of this application. Please refer to... Figure 9 The speech processing method in this embodiment may include the following steps S11 to S14.

[0136] S11. Obtain the user's preset feature information.

[0137] S12. Determine the target voice style for voice broadcasting based on the user's preset feature information.

[0138] S121. Select the appropriate document to be played based on the target speech style.

[0139] S13. Identify and extract the text content from the document to be played.

[0140] S14. Perform speech synthesis on the text content to generate a speech document with the target speech style.

[0141] The foregoing Figure 3 and Figure 8 The described embodiment does not require consideration of the compatibility between the document to be played and the user's current state (i.e., preset feature information). In this embodiment, after determining the target speech style, the appropriate document to be played is selected based on the user's current state. Since the user's current state can identify the target speech style, it can be regarded as selecting the appropriate document to be played based on the target speech style. Then, the document to be played is converted into a speech document with the target speech style through speech synthesis technology.

[0142] The implementation of step S13 includes at least one of the following two methods:

[0143] One method involves obtaining the preset feature information of a document based on its preset category tags, and identifying documents that match the target speech style obtained in step S12 as the documents to be played.

[0144] In the specific scenario of traditional e-publications, e-publications typically categorize all their documents and assign category tags, such as emotional, entertainment, humor, and horror. Based on this, appropriate preset feature information can be determined according to the category tags.

[0145] For documents to be played that have not been categorized, the terminal device can automatically determine their preset feature information based on the text content of the document, and then determine the appropriate voice style, which is another method described below.

[0146] Another approach is to determine the preset feature information of the document based on its text content, and then determine the appropriate speech style accordingly, identifying documents with the same speech style as the target document as the document to be played.

[0147] The technology of categorizing text content and generating category labels can be found in the section on display technologies. This method can be considered as requiring the terminal device to automatically generate category labels.

[0148] Figure 10 This is a schematic flowchart of the speech processing method according to the fourth embodiment of this application. Please refer to... Figure 10The speech processing method in this embodiment may include the following steps S11, S111, S112 and S12.

[0149] S11. Obtain the user's preset feature information.

[0150] S111. Obtain the preset feature information of the document to be played.

[0151] S112. Determine whether the preset feature information of the document to be played conflicts with the user's preset feature information.

[0152] If the preset feature information of the document to be played does not conflict with the user's preset feature information, it can be considered as compatible, and step S12 is executed. If there is a conflict, step S113 is executed.

[0153] S12. Determine the target voice style for voice broadcasting based on the user's preset feature information.

[0154] S113, Execute the preset strategy.

[0155] The foregoing Figure 9 The described embodiment selects a suitable document to be translated based on the target speech style. In contrast, this embodiment selects a suitable document to be translated based on preset feature information.

[0156] In step S111, obtaining the preset feature information of the document to be played includes at least one of the following:

[0157] One method involves obtaining the preset feature information of a document based on its preset category tags, and identifying documents that are the same as or similar to the preset feature information obtained in step S11 as documents to be played.

[0158] In the specific scenario of traditional e-publications, e-publications typically categorize all their documents and assign category tags, such as emotional, entertainment, humor, and horror. Based on this, appropriate preset feature information can be determined according to the category tags.

[0159] For documents to be played that have not been categorized, the terminal device can automatically determine their preset feature information based on the document's text content, which is another method described below.

[0160] Another approach is to determine the document's preset feature information based on its text content.

[0161] The technology of classifying and generating classification labels based on preset feature information can be found in the section on display technology. This method can be considered as requiring the terminal device to automatically generate classification labels.

[0162] In step S113, the preset strategy includes at least one of the following four:

[0163] The first method is to perform step S12.

[0164] The second method involves determining the target speech style based on the selection instructions.

[0165] When the preset feature information of the document to be played conflicts with the user's preset feature information, it can be considered that the voice style of the document to be played is not suitable for the user's current state. The user can issue a selection command to instruct the document to be played according to the voice style of the document to be played, or to play the document according to the voice style that is suitable for the user's current state (i.e., the target voice style), or to re-execute steps S111 and S112, that is, to re-select a document that is suitable for the target voice style, until the preset feature information of the selected document is the same as or similar to the user's preset feature information.

[0166] The third method involves determining the target speech style based on the preset feature information of the document to be played.

[0167] In other words, the speech style of the document to be played is used as the target speech style during speech playback.

[0168] The fourth option is to use the default voice style as the target voice style when playing audio.

[0169] The default voice style can be considered as the voice style that the terminal device defaults to before, and it has nothing to do with the voice style of the document to be played or the target voice style determined based on the user's current state.

[0170] In one implementation, the default voice style can be a traditional voice style that remains unchanged, or a pre-defined voice style that is not related to the user's state.

[0171] Figure 11 This is a schematic flowchart of the speech processing method according to the fifth embodiment of this application. Please refer to... Figure 11 The speech processing method in this embodiment may include the following steps S21 and S22.

[0172] S21. Obtain the preset feature information of the document to be played.

[0173] Preset feature information can be considered as information that indicates the document type, which can be used to identify the applicable user state. For example, the preset feature information of the document to be played includes, but is not limited to, at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age.

[0174] The specific content of the preset feature information of the document to be played, and the method of obtaining this preset feature information, can be found in the description of the foregoing embodiments, and will not be repeated here.

[0175] S22. Determine the target speech style for voice playback based on the preset feature information of the document to be played.

[0176] For a description of the principle of determining the target speech style based on preset feature information, please refer to the foregoing embodiments.

[0177] The difference from the previous embodiments is that this embodiment determines the target speech style based on preset feature information of the document to be played, rather than based on preset feature information of the user. The user can first select one or more documents as the documents to be played and obtain preset feature information from their text content. That is, the terminal device determines the target speech style based on the text content of the document to be played. Compared to determining the target speech style based on the user's current state, this embodiment can more realistically and accurately reflect the actual emotional expression of the document to be played, resulting in higher speech playback quality.

[0178] Figure 12 This is a schematic flowchart of the speech processing method according to the sixth embodiment of this application. Please refer to [link / reference]. Figure 12 The speech processing method in this embodiment may include the following steps S21 to S23.

[0179] S21. Obtain the preset feature information of the document to be played.

[0180] S211. Obtain the user's preset feature information;

[0181] S212. Determine whether the preset feature information of the document to be played conflicts with the user's preset feature information.

[0182] If the preset feature information of the document to be played does not conflict with the user's preset feature information, it can be considered as compatible, and step S22 is executed. If there is a conflict, step S23 is executed.

[0183] S22. Determine the target speech style for voice playback based on the preset feature information of the document to be played.

[0184] S23. Execute the preset strategy.

[0185] The user's preset feature information is used to identify the user's current state, and the preset feature information of the document to be played is used to identify the voice style applicable to the document itself. Therefore, this embodiment further determines whether the voice style applicable to the document itself matches the user's current state, which can make the voice playback more realistic and appropriate to the user's current state, and the voice playback quality is higher.

[0186] In step S23, the preset strategy includes at least one of the following four:

[0187] The first method is to perform step S22.

[0188] The second method involves determining the target speech style based on the selection instructions.

[0189] When the preset feature information of the document to be played conflicts with the user's preset feature information, it can be considered that the voice style of the document to be played is not suitable for the user's current state. The user can issue a selection command to instruct the document to be played according to the voice style of the document to be played, or to play the document according to the voice style that is suitable for the user's current state (i.e., the target voice style), or to re-execute step S21, that is, to reselect a document that is suitable for the user's current state, until the preset feature information of the selected document is the same as or similar to the user's preset feature information.

[0190] The third method involves determining the target speech style based on the user's preset feature information.

[0191] In other words, the user's voice style is used as the target voice style when playing the audio.

[0192] The fourth option is to use the default voice style as the target voice style when playing audio.

[0193] The default voice style can be considered as the voice style preset by the terminal device, which has nothing to do with the voice style of the document to be played or the voice style determined based on the user's current state.

[0194] In one implementation, the default voice style can be a traditional voice style that remains unchanged, or a pre-defined voice style that is not related to the user's state.

[0195] This application also provides a mobile terminal device, which includes a memory, a processor, and a voice processing program stored in the memory and executable on the processor. When the interactive program is executed by the processor, it implements the steps of the method in any of the above embodiments.

[0196] This application also provides a computer-readable storage medium storing a speech processing program, which, when executed by a processor, implements the steps of the speech processing method in any of the above embodiments.

[0197] The embodiments of the mobile terminal device and computer-readable storage medium provided in this application include all the technical features of the above-described method embodiments. The extended and explanatory content of the specification is basically the same as that of the above-described method embodiments, and will not be repeated here.

[0198] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the various possible embodiments described above.

[0199] This application also provides a chip including a memory and a processor. The memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, causing a device with the chip installed to perform the methods described in the various possible embodiments above.

[0200] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0201] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0202] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0203] The technical features of the present application can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present application.

[0204] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, controlled terminal, or network device, etc.) to execute the methods of each embodiment of this application.

[0205] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A speech processing method, characterized in that, The speech processing method includes: S11. Obtain the user's preset feature information, which is used to indicate the user's current state; Obtain the preset feature information of the document to be played; Determine whether the preset feature information of the document to be played conflicts with the user's preset feature information; If not, then execute S12 to determine the target voice style for voice broadcasting based on the user's preset feature information; if yes, then execute a preset strategy, which includes determining the target voice style based on the selection instruction and reselecting a document to be played that matches the target voice style until the preset feature information of the selected document to be played is the same as or similar to the user's preset feature information.

2. The method according to claim 1, characterized in that, Includes at least one of the following: The preset feature information includes at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age; The S11 step includes at least one of the following: obtaining preset feature information based on selection operation and / or input operation, obtaining preset feature information based on historical habits and / or sensors, obtaining user voice data, and obtaining preset feature information based on voice data.

3. The method according to claim 1, characterized in that, Includes at least one of the following: After step S12, the method further includes: identifying and extracting text content from the document to be played, and / or performing speech synthesis on the text content to generate a speech document with the target speech style; The method further includes: identifying and extracting the text content of the document to be played; after step S12, the method includes: performing speech synthesis on the text content to generate a speech document with the target speech style.

4. The method according to claim 1, characterized in that, The method further includes: The step of obtaining the preset feature information of the document to be played includes: obtaining its preset feature information based on the preset category tags of the document to be played, and / or determining its preset feature information based on the text content of the document to be played.

5. A speech processing method, characterized in that, The speech processing method includes: S21. Obtain the preset feature information of the document to be played; Obtain the user's preset feature information, which is used to indicate the user's current state, and determine whether the document to be played conflicts with the user's preset feature information; If not, then execute S22 to determine the target voice style for voice broadcasting based on the preset feature information of the document to be played; if yes, then execute the preset strategy, which includes reselecting a document to be played that is compatible with the user's current state until the preset feature information of the selected document to be played is the same as or similar to the user's preset feature information.

6. The method according to claim 5, characterized in that, Includes at least one of the following: The S21 step includes: obtaining its preset feature information based on the preset category tags of the document to be played, and / or determining its preset feature information based on the text content of the document to be played. The preset feature information includes at least one of the following: daily routine information, contextual information, emotional characteristics, personality characteristics, gender, and age.

7. A terminal device, characterized in that, The terminal device includes a memory and a processor. The memory stores a voice processing program, which, when executed by the processor, implements the steps of the voice processing method as described in any one of claims 1 to 6.

8. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the speech processing method as described in any one of claims 1 to 6.

Citation Information

Patent Citations

  • Voice processing method and device, client, terminal, server and storage medium

    CN108847214A