Method and apparatus for rendering audio data using adaptive fonts

Abstract

This disclosure relates to methods, devices, systems, and media for providing and using adaptive fonts for conveying the oral characteristics of speech contained in audio data in speech transcription. For example, a method may include acquiring audio data, acquiring text information transcribed from the speech contained in the audio data, determining a plurality of feature parameters associated with a plurality of speech features for the speech using a trained model, determining an adaptive font based on the plurality of feature parameters, and formatting at least a portion of the text information using the adaptive font for display to a user. The plurality of feature parameters may be specific to the speech. Various techniques for determining the adaptive font based on the plurality of feature parameters are disclosed.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to audiovisual processing, and in particular, to a technique for conveying oral characteristics of utterances contained in audio data in a transcript of the utterances, and related methods, apparatuses, systems, and media. BACKGROUND ART

[0002] Text is one of our most important means of communication, but since it only conveys information in characters, it is usually less suitable for conveying paralinguistic signals such as emotions than voice and / or images. This drawback becomes even more apparent when transcribing audio content (e.g., speech) into text and obtaining the transcript or translation, usually in plain text. A person reading subtitles can still experience the oral characteristics of the audio content of a movie, such as prosodic features (e.g., volume, speech rate, voice intonation), linguistic features (e.g., accent), and paralinguistic features (e.g., mood, temperament) by listening to the spoken utterance, but hearing-impaired or hard-of-hearing persons are deprived of these features when reading the same subtitles. When relying solely on plain text transcripts to understand audio content, important parts of communication can be systematically overlooked, which unduly affects and limits understanding and interaction with the surrounding environment and diverse audiovisual materials. SUMMARY OF THE INVENTION

[0003] The purpose of this disclosure is to enable full access to the original content of audio content, even in environments where access to audio content is unavailable or limited, or for the deaf or hard of hearing. That is, to enable access not only to the informational literal content of the audio information, but also to its prosodic features (such as volume, speech rate, and pitch), linguistic or phonetic content (such as language type and style, specific accent), and / or paralinguistic content (such as emotions like anger, joy, fear, and / or sadness). This is resolved by the subject matter of the separate claim.

[0004] Some embodiments of the present invention aim to enhance the transcription of audio content by automatically adapting the font used for the transcript, in particular to further convey the oral characteristics of the audio content that are typically missing in plain text transcripts. Such automatic font adaptation can be performed by automatically determining and using an adaptive font that visually adapts to the text information transcribed from the utterances contained in the acquired audio data, based on one or more determined phonetic aspects of the utterances. For example, multiple feature parameters related to multiple phonetic features can be determined for the utterance, and can be determined in particular using a trained model. Based on the multiple feature parameters, an adaptive font can be determined and used to format at least a portion of the text information. The formatted text information thus obtained can then be displayed to the user. For example, if the text information is a subtitle, such a formatted subtitle can be displayed simultaneously with the associated video.

[0005] A first aspect of this disclosure provides a computer implementation method which includes acquiring audio data, acquiring text information transcribed from utterances contained in the audio data, determining a plurality of feature parameters associated with a plurality of audio features for an utterance using a trained model, determining an adaptive font based on the plurality of feature parameters (where at least one aspect of the adaptive font is adapted based on at least one of the plurality of feature parameters), and formatting at least a portion of the text information for display to a user using the adaptive font.

[0006] In a preferred embodiment, the utterance is characterized in particular by a plurality of feature parameters, which may be specific to the utterance. For example, any of the feature parameters may correspond to an indicator of the relevance of its associated speech features to the utterance, for example, whether the associated speech features are particularly evident in the utterance.

[0007] In one embodiment, the feature parameters may include one or more paralinguistic parameters related to one or more paralinguistic features, one or more phonetic parameters related to one or more phonetic features, and / or one or more phonetic parameters related to one or more phonetic characteristics.

[0008] For example, one or more paralinguistic parameters may include one or more parameters related to one or more candidate emotions, each representing an estimate of whether the utterance expresses each candidate emotion.

[0009] Furthermore, in one embodiment, one or more phonetic parameters may be associated with one or more linguistic characteristics, including: a parameter indicating the detected language of the utterance; a parameter indicating the detected accent of the utterance; one or more confidence levels for one or more candidate languages ​​(each indicating a confidence level that the utterance is in each candidate language); and / or one or more confidence levels associated with one or more candidate accents (each indicating a confidence level that the utterance is in each candidate accent).

[0010] Furthermore, in another embodiment, one or more speech parameters related to one or more speech characteristics include one or more of the following: a parameter indicating the volume level of the utterance, a parameter indicating the speech rate of the utterance, and / or a parameter indicating the pitch of the utterance.

[0011] In a preferred embodiment, the feature parameters may be stored as multiple values ​​of a vector, preferably in an array or matrix.

[0012] According to one embodiment, the method may further include identifying one or more important speech features for an utterance based on a plurality of feature parameters, wherein the one or more important speech features and the one or more determined feature parameters in the plurality of feature parameters represent the utterance, and where determining an adaptive font based on the plurality of feature parameters includes determining an adaptive font based on the one or more determined feature parameters of the one or more important speech features.

[0013] According to this embodiment, multiple speech features can be further classified according to multiple speech feature categories, and identifying one or more important speech features for an utterance based on multiple feature parameters may include determining the dominant speech features in each of the multiple speech feature categories, and selecting one or more important speech features from the dominant speech features determined for the multiple speech feature categories.

[0014] Furthermore, in a preferred embodiment, one or more key speech features can be associated with one or more predetermined types of adaptations, and determining an adapted font based on one or more determined feature parameters of one or more key speech features may include applying one or more predetermined types of adaptations based on one or more determined feature parameters of one or more key speech features. Thus, it is possible to define dedicated types of adaptations that can be applied incrementally according to the determined feature parameters of one or more key speech features.

[0015] In a particularly preferred embodiment, the method further includes determining, for each of one or more important speech features, an application coefficient for a predetermined type of adaptation to the important speech feature based on determined feature parameters associated with the important speech feature. Here, applying one or more predetermined types of adaptations based on one or more determined feature parameters of one or more important speech features includes applying one or more predetermined types of adaptations to a predefined font in a proportion correlated with the application coefficient of one or more predetermined types of adaptations in order to generate an adaptive font. For example, this correlation may be positive, negative, linear, or logarithmic.

[0016] According to this embodiment, for each of one or more important speech features, the application coefficient of a predetermined type of adaptation can itself correlate with the determined feature parameter of the important speech feature. This may be a proportional correlation or other types of correlation. In one embodiment, this may include a mapping operation.

[0017] According to this embodiment, for each of the one or more important speech features, the application coefficient of each predetermined type of adaptation can lie on a scale having a minimum and a maximum value, where the minimum value represents a first degree (e.g., minimum degree) of the application of the predetermined type of adaptation, and the maximum value represents a second degree (e.g., maximum degree) of the application of the predetermined type of adaptation.

[0018] In this embodiment, the predefined font may be a variable font in which one or more embodiments can be progressively adapted. For example, one or more application coefficients for one or more predetermined types of adaptations may be one or more weights that influence the application of one or more predetermined types of adaptations. This embodiment enables the provision of particularly accurate extensions that faithfully reflect one or more determined feature parameters of one or more important speech features.

[0019] According to another embodiment, determining an adaptive font based on multiple feature parameters may include selecting one proxy font as the adaptive font from a plurality of proxy fonts based on the multiple feature parameters. For example, the plurality of proxy fonts may correspond to a plurality of predetermined fonts, and each font of the plurality of proxy fonts may be associated with a predetermined set of values ​​for the feature parameters for a plurality of speech features. Thus, in one embodiment, selecting one proxy font as the adaptive font from a plurality of proxy fonts based on multiple feature parameters may include selecting one proxy font from a plurality of proxy fonts whose predetermined set of values ​​is closest to the plurality of feature parameters.

[0020] Alternatively, in other embodiments, the multiple proxy fonts may correspond to a set of predetermined fonts generated for at least a range of discrete values ​​for each of one or more feature parameters of one or more important speech features, with each proxy font associated with a specific combination of discrete values ​​that form its respective set of proxy coordinates. In this case, selecting one proxy font as the adaptive font from the multiple proxy fonts based on multiple feature parameters may include selecting one proxy font from the multiple proxy fonts whose set of proxy coordinates is closest to the determined one or more feature parameters of one or more important speech features.

[0021] In yet another embodiment, determining an adaptive font based on multiple feature parameters may include generating an adaptive font based on multiple feature parameters using a further trained model. For example, the further trained model may be a model trained on a set of labeled fonts, each representing a specific speech feature, to determine reference associations between a set of combinations of values ​​for multiple feature parameters and each set of derived reference fonts. Using this further trained model, generating an adaptive font based on multiple feature parameters may include generating an adaptive font based on multiple feature parameters and reference associations.

[0022] Specifically, the step of generating an adaptive font based on multiple feature parameters and reference associations may include, in particular, applying the following to one or more fonts from a derived set of reference fonts: one or more adaptations of one or more microtypographical forms or one or more microtypographical elements; one or more adaptations of one or more macrotypographical forms or one or more macrotypographical elements; one or more adaptations of one or more display characteristics or one or more display elements; and one or more adaptations of one or more dynamic characteristics. According to this technique, a wide range of font adaptations can be used to generate particularly adaptive fonts.

[0023] In yet another embodiment, generating an adaptive font based on multiple feature parameters and reference associations may include generating the adaptive font based on multiple feature parameters and reference associations and using either a stable diffusion process or a character skeleton mapping process, thereby enabling accurate and rapid generation of the adaptive font.

[0024] In a further embodiment, determining an adaptive font based on one or more determined feature parameters of one or more important phonetic features may alternatively include applying one or more custom type adaptations to a custom font based on one or more determined feature parameters of one or more important phonetic features. For example, a custom font may correspond to a font customized to have a predefined appearance, particularly representing a group of people, an organization, a company, or a brand. In one embodiment, one or more custom type adaptations may be defined according to one or more custom rules for the custom font (e.g., embodied in a custom modification algorithm or rules developed in advance for the custom font). In one embodiment, one or more custom rules may be configured to restrict the application of one or more custom type adaptations in order to maintain the overall appearance of the custom font.

[0025] In many of the embodiments described above, predefined fonts, multiple proxy fonts, sets of labeled fonts, sets of derived reference fonts, or custom fonts may be obtained from a font source. Fonts provided by a font source may include fonts configured to represent specific phonetic features or combinations of phonetic features, for example, fonts configured to represent specific phonetic features or combinations of phonetic features. They may correspond to empirically developed fonts or fonts developed using artificial intelligence. In some examples, the font source may be a font repository, for example, font storage (local or remote).

[0026] In some of the foregoing embodiments, an utterance is a unit of speech (e.g., a voice statement or a sentence), and the plurality of feature parameters associated with the plurality of voice features determined for the utterance may correspond to a first plurality of feature parameters. The method may further comprise determining one or more speech subunits within the unit of speech that each have a respective second plurality of feature parameters that substantially deviate from the first plurality of feature parameters with respect to at least one voice feature, and determining a modified adaptive font for the one or more speech subunits based on the quantified deviation between the respective second plurality of feature parameters and the first plurality of feature parameters. Herein, formatting at least part of the text information based on the adaptive font for display to the user includes formatting one or more portions of the text information corresponding to the one or more speech subunits using the modified adaptive font. This particular embodiment may enable adaptation of the adaptive font at a specific granularity to reflect specific local variations in the unit of speech (e.g., emphasis on some words).

[0027] According to any of the above embodiments, the method may be adapted to convey oral characteristics of utterances of voice data, such as prosodic, linguistic and / or paralinguistic aspects, to a person with hearing impairment or hearing loss.

[0028] In a second aspect of the present disclosure, there is provided one or more computing devices comprising one or more processors configured to perform the method according to any embodiment of the first aspect of the present disclosure.

[0029] In a third aspect of the present disclosure, there is provided one or more computer-readable storage media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any embodiment of the first aspect of the present disclosure.

[0030] Many of the attendant features will be more readily appreciated as they become better understood by reference to the following detailed description considered in connection with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Certain features, aspects, and advantages of the present disclosure will be better understood with regard to the following description and accompanying drawings. [Figure 1] 1 illustrates an exemplary computing system according to an embodiment of the present disclosure. [Figure 2] 2 illustrates a font processing system according to an embodiment of the present disclosure. [Figure 3] 3 illustrates examples of a plurality of feature parameters according to an embodiment of the present disclosure. [Figure 4] 4 illustrates examples of visual adaptations applicable to generate an adaptive font according to an embodiment of the present disclosure. [Figure 5] 5 illustrates an example of encoding of an adaptive font determined for speech according to an embodiment of the present disclosure. [Figure 6] 6 illustrates an example of generating an adaptive font using a variable font generation technique according to an embodiment of the present disclosure. [Figure 7] 7 illustrates an example of generating an adaptive font using a discrete value generation technique according to an embodiment of the present disclosure. [Figure 8] 8 illustrates an example of generating an adaptive font using a custom font modification technique according to an embodiment of the present disclosure. [Figure 9] 9 illustrates an example of generating an adaptive font using an AI-based generation technique according to an embodiment of the present disclosure. [Figure 10] 10 shows a flow diagram of a method according to a preferred embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In the following description, drawings illustrating various embodiments are referenced. Various embodiments are also described below with reference to several examples. It should be understood that embodiments may include design and structural modifications without departing from the scope of the claimed subject matter. These examples are not intended to represent only the forms that may be constructed or utilized. However, identical or equivalent functions and sequences may be achieved by different examples. Furthermore, where used in this application and claims, the singular forms "a," "an," and "the" include the plural form unless the context clearly indicates otherwise. In addition, the term "includes" means "comprises." Furthermore, the term "combined" encompasses mechanical, electrical, magnetic, optical, and other practical methods of joining or linking items together, and does not exclude the presence of intermediate elements between combined items.

[0033] The techniques described herein can be implemented in a variety of computing environments. Examples are described in more detail below. Such environments generally involve the use of a well-configured computing device that implements multiple modules or units, each providing one or more operations necessary to complete the execution of such techniques. Each module or unit may be implemented in its own way; they do not all need to be implemented in the same way. Where used herein, a module or unit may be a structural component of a system that plays an operational role, which may be part or all of a software element (e.g., a function of a process, a discrete process, or any other suitable embodiment). A module or unit may contain computer executable instructions and may be encoded on a computer storage medium. Modules or units may be executed in parallel or serial, as appropriate, and may pass information to one another using shared memory on the computer on which they are running, using message-passing protocols, or in any other suitable way.

[0034] Figure 1 shows a computing system 100 of interest according to some embodiments of the present disclosure, which is configured to process at least audio data 101 and text information 105 that transcribes the utterances contained in the audio data 101 to provide formatted text information 107 that is at least partially formatted using an adaptive font determined based on the oral characteristics associated with the utterances. Furthermore, in some embodiments, the computing system further comprises an output device 113 for displaying the formatted text information 107 and, for example, the audio data 101 to a user.

[0035] For example, computing system 100 may comprise a font processing system 111, an output device 113, a control unit 115, and optionally a transcription unit 121 and an audio / audiovideo processing unit 117. For example, computing system 100 may be part of a system for providing audiovideo content to a user for consumption. In one example, audiovideo data is received (e.g., as a stream). Audiovideo data can be any kind of audiovideo content, such as any kind of video (e.g., movies), online streams, video conferencing, audiovideo chat, video games, etc. For example, audiovideo data may be received as one or more input signals including a signal carrying video data 103 (e.g., a video data stream) and a signal carrying corresponding audio data 101 (e.g., an audio data stream). In some embodiments, audiovideo data may further include text information 105 that transcribes the informational content of the audio data 101 in writing, such as subtitles or closed captions. In another embodiment, an optional transcription unit 121 may be provided to generate the text information 105 from the audio data 101. In one embodiment, the text information 105 may be received and / or generated at least in synchronization with, or nearly in synchronization with, the audio data 101. For example, it may correspond to a separate stream associated with the audio-video data.

[0036] In one embodiment, the computing system 100 is configured to identify and acquire text information 105 that transcribes the / encoded utterances contained in the audio data 101, and to provide the audio data 101 containing the utterances and the text information 105 to the font processing system 111 in a substantially synchronous manner. The font processing system 111 may then be configured to process them to output formatted text information 107 corresponding to the utterances. In one example, the formatted text information 107 may correspond to an output stream provided to the output device 113 for output to the user, associated with at least the audio data 101 and video 103. In one example, this output stream may be associated with the audio data 101 and video data 103 in a timely manner.

[0037] The output device 113 can be any type of output device capable of rendering audio-video data (including audio data 101 and video data 103) and associated formatted text information 107 (e.g., as subtitles). In one example, at least the video data 103 and formatted text information 107 may be rendered to the user by the output device 113. In another example, the complete audio-video content of the audio-video data, including audio data 101 and video data 103, may be rendered by the output device 113 along with the formatted text information 107. For example, the output device 113 may include a display device, a mobile device, a tablet device, a laptop, a projection device (such as a beamer), and optionally include any associated audio output device for the audio data 101.

[0038] In some embodiments, the computing system 100 includes a control unit 115 configured to control at least a font processing system 111. For example, the control unit 115 may be configured to receive optional configuration inputs and / or calibration inputs (not shown) for configuring and / or calibrating the font processing system 111. In one embodiment, the control unit 115 may receive configuration inputs from a user, for example, via a configuration menu displayed to the user using an output device 113. The configuration inputs may provide user settings that configure the extensions provided by the adaptive font of the formatted text information 107. For example, the configuration inputs may include a setting indicating whether the user desires extensions associated with speech characteristics, linguistic characteristics and / or paralinguistic aspects detected in the speech data. The font processing system 111 is configured to adapt the formatted text information 107 delivered accordingly.

[0039] In some embodiments, the font processing system 111 and the control unit 115 belong to the processing system 119. In some examples, the processing system 119 and the output device 113 may belong to the same computing device, e.g., a user computing device (e.g., a tablet, personal computer, laptop, or mobile device). In other examples, the processing system 119 and the output device 113 may belong to separate computing devices that communicate with each other. For example, the processing system 119 may be part of a remote computing device (e.g., a cloud computing system) and communicate with a local / user computing device comprising the output device 113 via a network.

[0040] Furthermore, the processing system 119 may optionally include an audio / audiovideo processing unit 117 for converting the audio data 101 and / or video data 103 into an input format supported by the output device 113. Alternatively, the audio / video data may already be provided in a format suitable for the output device 113. For example, the output device 113 may include separate processing means for processing the audio data 101 and / or video data 103 for rendering to the user.

[0041] In an alternative embodiment, the computing system 100 may be provided without video data and video processing means. For example, such a computing system may be particularly suited to processing audio streams (e.g., podcasts), audio broadcast data, audio data of calls (e.g., telephone calls), audio data of events currently being experienced by a user, such as live events (e.g., conferences being received by a user using an audio processing / recording device) or virtual events. In such a case, the computing system 100 may simply receive audio data 101 and optionally text information 105 as input and, since it does not use video signals, may remain entirely the same as described above. Such an alternative computing system 100 could correspond, for example, to the computing system 100 in Figure 1 without traced video data 103. In such a case, the output device 113 may also be an arbitrary output device having a display for displaying formatted text information 107 to the user. For example, it may be included in a user computing device (e.g., a mobile device).

[0042] In any embodiment of the computing system 100, the audio data 101 can be received, for example, in WAV format.

[0043] Figure 2 shows a font processing system 200 that can correspond to the font processing system 111 of the computing system 100 in Figure 1. In one embodiment, it comprises a speech analysis unit 217, an adaptive font generation unit 219, a font source 223, and a font application unit 221. In one embodiment, the speech analysis unit 217 takes audio data 201 containing utterances (e.g., audio data 101 in Figure 1) and text information 205 transcribed from the utterances (e.g., text information 105 in Figure 1) as input and is configured to determine a plurality of feature parameters 211 associated with a plurality of speech features for the utterances using a trained model. Based on the obtained plurality of feature parameters 211, the adaptive font generation unit 219 is configured to determine an adaptive font 213, where at least one aspect of the adaptive font is adapted based on at least one of the plurality of feature parameters. In some embodiments, as described below, this can further be based on one or more fonts 215 provided by the font source 223. Next, the font application unit 221 is configured to format at least a portion of the text information 205 using an adaptive font 213 for display to the user. In some examples, the output of the font application unit 221 may be formatted text information 107 associated with the utterance, which is then provided to the output device 113 for display to the user.

[0044] In some preferred embodiments, the speech analysis unit 217 is responsible for determining a plurality of feature parameters 211 associated with a plurality of speech features of an utterance, based on audio data 201 containing the utterance. In one embodiment, the utterance may correspond to a speech statement or expression contained in the audio data 201. It may be associated in particular with a person (e.g., a speaker, singer, or character speaking in the audio data). For example, it may correspond to a sentence. In certain cases, it may correspond to a single spoken word (e.g., an interjection). Furthermore, a specific time window of the audio data 201 may be associated with the utterance. The same time window may also be associated with a plurality of feature parameters determined for the utterance. In some embodiments, an intermediate processing unit (not shown) may be provided in the font processing system 200 (e.g., the speech analysis unit 217) to associate one or more utterances identified in the audio data 201 with one or more text information 205 that transcribes one or more utterances.

[0045] According to some preferred embodiments, the speech analysis unit 217 is configured to evaluate, for each of a plurality of speech features, whether the utterance exhibits that speech feature and / or the quantifiable result / confidence level in the evaluation. Thus, the plurality of speech features can correspond to a plurality (e.g., a list or catalog) of speech features that the utterance potentially exhibits. In some embodiments, the evaluation results are provided as a plurality of feature parameters associated with the plurality of speech features. These plurality of feature parameters can specifically characterize the utterance in terms of, for example, values, cues, estimates, etc.

[0046] For example, the speech analysis unit 217 can provide corresponding quantification results representing the evaluation results for each given speech feature of a plurality of speech features. For a particular type of speech feature, the quantification results may be probabilities, confidence levels, percentage values, etc. The quantification results can then be stored as feature parameters associated with the given speech feature. Thus, the speech analysis unit 217 is configured to repeat such evaluations for each of the plurality of speech features in order to determine a plurality of feature parameters 211 associated with the plurality of speech features of an utterance.

[0047] In some embodiments, the trained model of the speech analysis unit 217 may include or correspond to a speech emotion recognition model. For example, it may include a machine learning model, a deep learning model, a large-scale language model, and / or a natural language processing model.

[0048] In one embodiment, a speech emotion recognition model can be provided or generated as follows: First, a model type, or combination of models, is selected (see the exemplary list above). Next, the selected model is trained on labeled training data, for example, using a supervised learning process. Third, the model can be fine-tuned (for example, using hyperparameter tuning). Fourth, a cross-validation process is performed using samples from different domains and / or contexts. Fifth, the resulting trained model can be packaged, for example, as an application program interface, API, for use. In some examples, domain knowledge can be used to define specific associations.

[0049] In one embodiment, the speech analysis unit 217 is configured to determine a number of feature parameters 211 for an utterance using a trained model (e.g., a speech emotion recognition model provided as an API) as follows:

[0050] First, a portion of the audio data 201 associated with the utterance can be identified and processed. In some cases, one or more intermediate data transformations may be required.

[0051] Next, one or more audio features can be extracted from a portion of the audio data, such as Mel-frequency cepstrum coefficients (MFCCs), zero crossing ratios (ZCRs), chromagram short-time Fourier transform values ​​(Chroma-STFTs), root mean square values ​​(RMS), and / or Mel-scale spectrograms (MSS).

[0052] Next, a portion of the audio data and one or more audio features are input into the trained model.

[0053] Furthermore, based on a portion of the audio data and one or more audio features, the trained model determines multiple feature parameters 211 associated with multiple audio features.

[0054] In preferred embodiments of this disclosure, the multiple speech features considered by the trained model may belong to one or more speech feature categories. For example, they may be from multiple speech feature categories such as paralinguistic feature categories, phonetic feature categories and / or speech characteristic categories. Thus, the multiple feature parameters 211 obtained as a result of a given utterance may include: For example, in a paralinguistic feature category, one or more paralinguistic parameters associated with one or more paralinguistic features, For example, in a phonetic feature category, one or more phonetic parameters associated with one or more phonetic features, and / or For example, in a voice characteristic category, one or more voice parameters associated with one or more voice characteristics.

[0055] In some preferred embodiments, one or more paralinguistic parameters may include one or more parameters associated with one or more candidate emotions, each representing an estimate of whether the utterance exhibits each candidate emotion. The one or more candidate emotions may include one or more such as anger, joy, fear, sadness, vigor, and harmony.

[0056] In some preferred embodiments, one or more phonetic parameters may relate to linguistic characteristics, in particular, one or more of the following: a parameter indicating the detected language of the utterance; a parameter indicating the detected accent of the utterance (e.g., including social dialects and dialects); one or more confidence levels for one or more candidate languages ​​(each indicating a confidence level that the utterance is in its respective candidate language); and / or one or more confidence levels associated with one or more candidate accents (each indicating a confidence level that the utterance is in its respective candidate accent).

[0057] In some preferred embodiments, one or more speech parameters associated with one or more speech characteristics may include one or more of the following: a parameter indicating the volume level of the utterance, a parameter indicating the speech rate of the utterance, and / or a parameter indicating the pitch of the utterance. It may also include one or more parameters representing rhythm, quality, the percentage of pauses in the utterance, etc.

[0058] Figure 3 provides an example of multiple feature parameters 301 (e.g., 211) provided for a given utterance. In one example, the multiple feature parameters 301 can be associated with a specific time window or sequence of audio data 201 for clear association with the analyzed portion of the audio data 201 containing the utterance. In this example, the multiple feature parameters 301 include a paralinguistic feature category of “emotion,” which includes four candidate emotions “hatred,” “joy,” “fear,” and “sadness,” each associated with its respective paralinguistic parameter (“0.1,” “0.0,” “0.8,” and “0.1”). The multiple feature parameters 301 may further include a “speech” speech feature category, which includes a “volume” parameter (“0.5”) indicating the volume level of the utterance, a “velocity” parameter (“0.5”) indicating the speech rate of the utterance, and a “pitch” parameter (“0.7”) indicating the pitch of the utterance. Pitch parameters may be particularly relevant to distinguish the gender and / or type of speaker or singer (e.g., adult, child, etc.) and / or to identify specific changes in intonation or stress in the audio data 201. Multiple feature parameters 301 may also include phonetic feature categories of “language,” including a parameter indicating the detected language of the utterance (“English”) and a parameter indicating the detected accent of the utterance (“British”).

[0059] In a preferred embodiment, the multiple feature parameters 301, as shown in Figure 3, can be at least partially normalized and stored as multiple values ​​(e.g., coordinates) of a vector. They can be stored in an array or matrix.

[0060] Returning to Figure 2, the adaptive font generation unit 219 is configured to determine the adaptive font 213 that best represents the utterance based on several feature parameters 301. When applied to the text information 205 associated with the utterance by the font application unit 221, the resulting formatted text information 107 conveys not only the plain text of the text information 205 but also at least some of the most important oral characteristics of the utterance using the modified visual aspects or features of the adaptive font 213. By using the adaptive font 213, there is no need to display additional visual effects to visually represent or convey these characteristics. The user simply reads the text to obtain the literal content, enhanced with the corresponding font adaptation, all at once.

[0061] Figure 4 shows an example of such an extension, displaying an image of video 403 shown on display device 401, in which character 405 is speaking and uttering utterances. Element 407 shows an example of formatted text information (e.g., subtitles) that is typically associated with utterances. In this example, element 407 simply illustrates various ways of extending text information to convey paralinguistic or prosodic features using adaptive fonts.

[0062] Specifically, the plain text content (in German) of the text information is as follows: “Die Schrift wird “dicker” und “dunner”, “breiter” und “schmaler”, “angstlich”, “wutend” und mehr”. (The font can become "bold" or "thin," "wide" or "narrow," and can appear "anxious" or "angry.")

[0063] This means the following in English: “The font becomes “thicker” and “thinner”, “wider” and “narrower”, “fearful”, “angry” and more”. (The font can be "bolder," "thinner," "wider," "narrower," "fear," "anger," etc.)

[0064] However, in this example, the font will look like this: To indicate a high volume level of speech, the sound is made thicker (dicker in German), To indicate a low volume level of speech, make it thinner (like the German word "dunner") To indicate a low speech rate, more broadly (in German, "breiter"), To indicate a high rate of speech, the mouth is narrower (in German, "schmaler") A leftward tilt (see the word "angstlich," i.e., fear) indicates a generally passive feeling detected in the emotion or utterance of fear, or A rightward tilt indicates an overall active emotion detected in the emotion or utterance of anger (see the word "wutend," i.e., anger).

[0065] These are merely illustrative examples. The expected font adaptations are not limited to these examples and may include combinations of adaptations based on one or more predefined fonts, and / or one or more of the following modifications: Microtypographic aspects or elements (e.g., serifs, stroke width, font width, ascenders, descenders, type orientation, and vertical orientation of one or more characters of a predefined font), one or more macrotypographic aspects or elements (e.g., font size, character spacing, line spacing, word rotation, word transparency, and word emphasis of a predefined font), one or more display characteristics or elements (e.g., one or more font colors, highlight colors, or shadowing effects of a predefined font), and / or one or more dynamic characteristics (e.g., providing additional vibration effects or specific animations to adaptive fonts).

[0066] A non-exhaustive list of exemplary extensions is provided below. For example, determining an adaptive font 213 based on multiple feature parameters 211 may include one or more of the following: Applying a predefined font typeface to bold when one or more feature parameters from multiple feature parameters indicate a high volume level of speech, and / or applying a predefined font typeface to light when one or more feature parameters from multiple feature parameters indicate a low volume level of speech. If one or more feature parameters from multiple feature parameters indicate a high speech rate, the predefined font character spacing and / or character width is reduced, and / or if one or more feature parameters from multiple feature parameters indicate a low speech rate, the predefined font character spacing and / or character width is increased. Tilting a predefined font orientation in the direction corresponding to the reading direction when one or more feature parameters from multiple feature parameters represent a significant presence of activity in the utterance, and / or tilting a predefined font orientation in the direction opposite to the reading direction when one or more feature parameters from multiple feature parameters represent a significant presence of passivity in the utterance. Adapting the vertical orientation of one or more characters of a predefined font by turning the ends of one or more characters upward when one or more feature parameters from multiple feature parameters represent a significant presence of joy or a positive emotion in the utterance, and / or adapting the vertical orientation of one or more characters of a predefined font by turning the ends of one or more characters downward when one or more feature parameters from multiple feature parameters represent a significant presence of sadness or a negative emotion in the utterance. To express harmony, for example, the axis of the shadow is directed at 45 degrees, or to express disharmony, an unusual placement of the shadow axis at more than 90 degrees is used. The use of sharp, angular shapes and curves to express hard, consonant, and plosive speech or negative emotions. The use of rounded shapes and curves to express soft, vowel-heavy speech or positive emotions. To represent balance, use contrast with uniform stroke widths, or to represent imbalance, use contrast with high stroke widths. To reflect the speaker's emotional stability, use the static appearance of characters. The use of glyph ends pointing in the corresponding direction to represent deviations from the standard of sound, and Mapping the spacing between characters to a measure of sound density.

[0067] Figure 5 shows a specific representation of the adaptive font 213 determined for a particular utterance (e.g., the sentence "Hurry up Steve!"). For example, based on several feature parameters 301 shown in Figure 3, the adaptive font generation unit 219 can be configured to determine an adaptive font 213 that includes values ​​or parameters representing microtypographic aspects, macrotypographic aspects, display characteristics, dynamic characteristics, etc., as shown in Figure 5.

[0068] In a preferred embodiment, the font processing system 200, and in particular its adaptive font generation unit 219, can be configured to first identify one or more significant speech features for an utterance based on a plurality of feature parameters 211, where one or more significant speech features and their respective determined feature parameters in the plurality of feature parameters 211 represent the utterance. In this case, determining the adaptive font based on the plurality of feature parameters 211 may include determining the adaptive font based on one or more determined feature parameters of one or more significant speech features. In other words, in some embodiments, particular focus is placed on the most prominent speech features detected for the utterance from among the plurality of speech features.

[0069] In one embodiment, multiple phonetic features are classified according to multiple phonetic feature categories, and identifying one or more important phonetic features for an utterance based on multiple feature parameters 211 may include determining a dominant phonetic feature in each of the multiple phonetic feature categories, and selecting one or more important phonetic features from the dominant phonetic features determined for the multiple phonetic feature categories. For example, a dominant phonetic feature may be identified in a paralinguistic feature category (e.g., anger), and another dominant phonetic feature may be identified in a phonetic feature category (e.g., French accent).

[0070] In some embodiments, one or more key speech features can be associated with one or more predefined adaptive types, and determining an adaptive font based on one or more determined feature parameters of one or more key speech features can include applying one or more predefined adaptive types based on one or more determined feature parameters of one or more key speech features. For example, each of the one or more key speech features can be associated with a particular type of font adaptive, such as those listed above (see non-exhaustive list).

[0071] In a particularly preferred embodiment, the adaptive font generation unit 219 may include or be embodied as a variable font generation unit 219a (see Figure 2) configured to determine an adaptive font 213 based on a plurality of feature parameters 211 using variable font generation technology. For example, for each of one or more important speech features, it may be included to determine an application coefficient of a predefined adaptive type to the important speech feature based on the determined feature parameters associated with the important speech feature. In this case, applying one or more predefined adaptive types based on one or more determined feature parameters of one or more important speech features may include applying one or more predefined adaptive types to a predefined font at a rate that correlates with the application coefficient of one or more predefined adaptive types in order to generate an adaptive font. This can provide a gradual effect in the application of each type of adaptation to the predefined font. The predefined font may correspond to a font provided by a font source 223. For example, it may correspond to a particularly easy-to-read basic font.

[0072] Depending on the context, the degree of correlation can be positive, negative, linear (proportional), or logarithmic.

[0073] In certain cases, for each of one or more important speech features, the application coefficient of each predefined adaptive type can be correlated with the determined feature parameters of the important speech features. This correlation can be linear or follow any specific formula. This correlation can also be implemented as a mapping.

[0074] In these cases, for each of the one or more important speech features, the application coefficient of each predefined adaptive type may lie on a scale (e.g., an axis) having a minimum and a maximum value, where the minimum value represents a first degree of application of the predefined adaptive type, and the maximum value represents a second degree of application of the predefined adaptive type. The first degree of application of the predefined adaptive type may be the minimum degree of application of the predefined adaptive type, the second degree of application of the predefined adaptive type may be the maximum degree of application of the predefined adaptive type, and vice versa. In particular cases, the first degree of application of the predefined adaptive type may be the maximum negative degree of application of the predefined adaptive type (e.g., the maximum negative slope), and the second degree of application of the predefined adaptive type may be the maximum positive degree of application of the predefined adaptive type (e.g., the maximum positive slope). In some preferred embodiments, the scale is a normalized scale (e.g., between 0 and 1, or between -1 and +1). Additionally or alternatively, the median of the scale may correspond to the value of no application of the predefined adaptive type. For example, if the scale ranges from -1 to +1, a value of 0 can correspond to a neutral position, i.e., the non-application of a given type of adaptation to a given font.

[0075] According to this variable font generation technique, a predefined font may be a variable font in which one or more aspects are progressively adaptable or parameterized. In one embodiment, one or more application coefficients for one or more predefined adaptation types are one or more weights that affect the application of one or more predefined adaptation types.

[0076] For example, consider a case where a key speech feature is particularly loud, as shown in Figure 4. This may mean that multiple feature parameters include parameters associated with loudness speech characteristics that exhibit high loudness levels (e.g., exceeding a predetermined threshold) in multiple feature parameters 211. Adaptation of a particular type of predefined font may be associated with loudness speech characteristics, and the typeface of the predefined font may be adapted to be bold or thicker when parameters associated with loudness speech characteristics exhibit high loudness levels.

[0077] Furthermore, or alternatively, the same utterance may possess key vocal features of an utterance that indicates an angry mood. This may mean that multiple feature parameters 211 may include parameters associated with a high (e.g., exceeding another predetermined threshold) candidate anger emotion. Adaptations of certain types of predefined fonts may also be associated with a candidate anger emotion, for example, by tilting the orientation of the predefined font in a direction corresponding to the reading direction.

[0078] In this example, the variable font generation unit 219a may be configured to adapt the typeface of a predefined font to be bold and / or tilt the orientation of the predefined font in a direction corresponding to the reading direction. Furthermore, based on the determined feature parameters in multiple feature parameters 211, respective application coefficients may be determined for the loudness speech characteristic and the candidate anger emotion. These application coefficients may correspond to values ​​on their respective scales (e.g., coordinates), and each type of adaptation may be applied at a ratio (e.g., proportionally) that correlates with these application coefficients. Thus, the bolding effect applied to the predefined font may be progressive or variable, so as to progressively represent the values ​​of the parameter associated with the loudness speech characteristic. Similarly, the tilt of the orientation of the predefined font in the reading direction may correlate with the value of the parameter associated with the candidate anger emotion. For example, if this determined feature for the candidate anger emotion indicates an 80% confidence level that the utterance indicates anger, then the applied tilt of the orientation of the predefined font in the direction corresponding to the reading direction may correspond to 80% of the maximum tilt.

[0079] Figure 6 specifically illustrates the variable font generation technique described above. For example, imagine that several important sound features are identified as liveliness, loudness level, and harmony. In one embodiment, the determined feature parameters of several feature parameters 211 may be -0.6, 0.8, and -0.6, respectively. Furthermore, a first type of adaptation for liveliness may be the tilt of the predefined font's orientation, a second type of adaptation for loudness may be the thickness of the character strokes, and a third type of adaptation for harmony may be the left-right balance of the stroke thickness. Thus, considering the linear correlation between these determined feature parameters and the respective application coefficients for these types of adaptations, the application coefficients may correspond equally to -0.6 for the first type of adaptation, 0.8 for the second type of adaptation, and -0.6 for the third type of adaptation. All these types of adaptations may be collectively applied to the predefined font, resulting in the adapted font shown in the lower right corner of the figure.

[0080] In another embodiment, the adaptive font generation unit 219 may comprise, or be embodied as, a discrete value font generation unit 219b (see Figure 2) configured to determine the adaptive font 213 based on a plurality of feature parameters 211 using discrete value generation techniques. This may include selecting one proxy font as the adaptive font 213 from a plurality of proxy fonts based on the plurality of feature parameters 211. For example, the font source 223 may be configured to provide several prepared fonts, each representing a plurality of types of phonetic features or combinations of types of phonetic features.

[0081] For example, a particular font may be predetermined for a specific combination of speech features (e.g., paralinguistic speech features) to form a plurality of proxy fonts. The fonts of the plurality of proxy fonts may each be associated with a predetermined set of values ​​of feature parameters for a plurality of speech features. Thus, in this embodiment, the discrete-value font generation unit 219b may be configured to select from the plurality of proxy fonts the proxy font whose predetermined set of values ​​is closest to the plurality of feature parameters 211. This selection may be performed, for example, by determining the distance (e.g., Euclidean distance) between a first vector containing the plurality of feature parameters 211 associated with an utterance and each of the plurality of predetermined vectors associated with the plurality of proxy fonts. Other decision techniques are also considered.

[0082] Alternatively, multiple proxy fonts may correspond to at least several predetermined fonts generated for a range of discrete values ​​for each of one or more feature parameters of one or more important speech features, with each proxy font associated with a particular combination of discrete values ​​that form each set of proxy coordinates. For example, as seen in Figure 7, element 705 may correspond to such multiple proxy fonts. This may include different proxy fonts for at least various discrete values ​​of feature parameters for the speech features of liveliness, loudness level, and harmonicity. Each proxy font may correspond to a corresponding ratio of a corresponding type of adaptation applied to a particular predefined font, for example using the variable font generation technique described above, or to a particular font specially developed for a combination of discrete values ​​(as shown in element 705). For example, multiple proxy fonts may include proxy fonts for discrete values ​​of feature parameters for liveliness, loudness level, and harmonicity ranging from (-1, 0, -1) to (+1, 1, +1), with, for example, a discrete step of 0.25. In this example, the step of selecting a proxy font as an adaptive font from multiple proxy fonts based on multiple feature parameters 211 may involve selecting a particular proxy font from multiple proxy fonts whose set of proxy coordinates is closest to one or more determined feature parameters of one or more important speech features. For example, for one or more determined feature parameters of one or more important speech features having values ​​(-0.6, 0.8, -0.6) for liveliness, loudness level, and harmonicity (see element 701), the discrete-value font generation unit 219b may select a proxy font with proxy coordinates (-0.5, 0.75, -0.5) in 705 and use it as an adaptive font (see 707).

[0083] One advantage of discrete-value font generation techniques is that they require fewer computational resources because they only require a mapping process instead of the parameterization process of variable fonts. This can potentially accelerate adaptive font generation.

[0084] In yet another embodiment, the adaptive font generation unit 219 may include, or be embodied as, a customized font modification unit 219c (see Figure 2) configured to determine the adaptive font 213 based on a plurality of feature parameters 211 using custom font modification techniques. This may include applying one or more custom types of adaptations to the custom font based on one or more determined feature parameters of one or more important speech features. For example, one or more custom types of adaptations may be defined according to one or more custom rules for the custom font.

[0085] For example, a custom font may correspond to a font customized to have a predefined appearance. This may represent, for example, a group of people, an organization, a company, or a brand, or be infamously associated with one. Its use may be restricted. In this embodiment, one or more custom rules may be implemented in the form of a modification algorithm or correspond to a pre-developed, predefined custom adaptation. In some embodiments, this may involve applying several types of adaptations similar to those described above in variable font generation techniques, for example, using incremental adaptations that correlate with feature parameters of several key vocal features identified in an utterance. Figure 8 shows such pre-determined custom adaptations for different custom fonts 803, 805. Starting with different custom fonts, different adapted fonts 807, 809 are obtained for the same feature parameter 801 of key vocal features 801 for loudness level, vigor, and anger.

[0086] In one embodiment, one or more custom rules may be configured to restrict the application of one or more custom types of adaptations in order to maintain the overall appearance of the custom font, that is, to maintain the association between the adapted custom font and, for example, its associated organization or company.

[0087] In a further embodiment, the adaptive font generation unit 219 may comprise or be embodied as an artificial intelligence (AI)-based font generation unit 219d (see Figure 2), configured to determine an adaptive font 213 based on a plurality of feature parameters using AI-based generation techniques. This may include using a further trained model to generate the adaptive font 213 based on a plurality of feature parameters 211.

[0088] For example, a further-trained model might be one trained with a set of labeled fonts, each representing a specific speech feature. The further-trained model might be trained to learn several reference associations between sets of value combinations for multiple feature parameters 211 and each set of derived reference fonts. Once trained, adaptive font generation using the further-trained model might be based on the multiple feature parameters 211 and their reference associations.

[0089] Furthermore, generating adaptive fonts based on multiple feature parameters 211 and reference associations may, in particular, involve applying one or more adaptations of one or more microtypographic aspects or one or more microtypographic elements, one or more adaptations of one or more macrotypographic aspects or one or more macrotypographic elements, one or more adaptations of one or more display characteristics or one or more display elements, and / or one or more adaptations of one or more dynamic characteristics to one or more fonts from a set of derived reference fonts (see also the non-exhaustive list above). AI-based generation techniques are particularly well-suited to integrating or developing various types of adaptations that represent specific speech features or specific combinations of speech features recognized in utterances.

[0090] Furthermore, the step of generating an adaptive font based on multiple feature parameters 211 and reference associations may further include either a stable diffusion process or a character skeleton mapping process. Details are shown in Figure 9 below.

[0091] For example, a preliminary design analysis process 901 may first be performed as part of training a further trained model. Multiple labeled fonts 902 are provided (e.g., from a font source 223). Some of them may be associated with several representative combinations of feature parameters of multiple speech features. Some of them may, alternatively or additionally, simply specifically represent some speech features (e.g., some particular emotions). Feature extraction 903 may be performed on the multiple labeled fonts 902 using deep learning or machine learning 905, and a classification process 907 may be further performed to learn several reference associations 909 between sets of value combinations for multiple feature parameters of the feature parameters and each set of derived reference fonts. For example, this may correspond to a language-design link assignment 909 where diverse types of languages ​​are associated with each combination of their respective designs, derived reference fonts, and / or types of adaptations (e.g., combinations of microtypographic adaptations).

[0092] According to the first implementation, the further trained model may be configured to determine an adaptive font 933 (e.g., 213) using a stable diffusion process 711. For example, a preliminary diffusion process 913 may be performed on multiple labeled fonts 902. Furthermore, semantic mapping 915 may be performed based on multiple feature parameters 211 determined for the utterance. Subsequently, a pre-trained design generation may be performed using a despreading process 917, typically yielding a pixelated intermediate-generated font 919. A vectorization process 921 on the resulting intermediate-generated font 919 can yield an adaptive font 933, e.g., 213 in Figure 2 or Figure 5.

[0093] According to a second implementation of this example, skeleton mapping 923 may be performed using a further trained model and reference association 909. This may include a skeleton adjustment step 925, a stroke application step 927, and a style adjustment step 929. The result may be a font 933, for example, an adaptive font 213.

[0094] Furthermore, in either implementation, a metrics determination and kerning process 935, as well as some design adjustments 937, may follow to obtain the final text output 939 (e.g., subtitle output). These steps may be performed by the font application unit 221 in particular, as shown in Figure 2.

[0095] Returning to Figure 2, the font application unit 221 is configured to format at least a portion of the transcribed speech text information 205 for display to the user using the adaptive font 213 determined by the adaptive font generation unit 219. For example, a portion of the transcribed speech text information 205, or the entirety of the text information 205, may be formatted by uniformly applying the adaptive font 213. This may include additional steps of metrics determination, kerning, and / or design adjustments. The output of the font application unit 221 may be the formatted text information 207 (also referred to as 107 in Figure 1).

[0096] In yet another embodiment, the utterances analyzed by the font processing system 200 may correspond to utterance units that typically correspond to sentences or spoken texts. However, in some embodiments, the adaptive font 213 determined for an utterance does not necessarily have to be applied uniformly to the text information 205 that transcribes the utterance.

[0097] For example, a plurality of feature parameters 211 associated with a plurality of speech features determined for an utterance may correspond to a first plurality of feature parameters. The front processing system 200 (e.g., a speech analysis unit 217) may be further configured to determine one or more utterance subunits within an utterance unit, each having a second plurality of feature parameters that substantially deviate from the first plurality of feature parameters with respect to at least one speech feature. The front processing system 200 (e.g., an adaptive font generation unit 219) may further determine a modified adaptive font for one or more utterance subunits based on the quantified deviation between each of the second plurality of feature parameters and the first plurality of feature parameters. Finally, the front processing system 200 (e.g., a font application unit 221) may be configured to format one or more portions of the text information 205 corresponding to one or more utterance subunits using the modified adaptive font, while maintaining the formatted state of the remaining portion of the text information 205 using, for example, the adaptive font 213. In this way, certain local paralinguistic or phonetic deviations in an utterance, such as emphasis on words or terms, may be rendered optimally for the user.

[0098] According to a particular embodiment, formatting at least a portion of the text information 205, which is a transcription of an utterance for display to a user, using an adaptive font 213 determined by an adaptive font generation unit 219 may further include adding one or more specific terms or expressions to the text information that describe several oral characteristics characterized by a plurality of feature parameters. For example, one or more specific terms or expressions may correspond to added closed captions that describe in text a paralinguistic feature (e.g., laughter) detected in the utterance. One or more specific terms or expressions may be formatted using an adaptive font.

[0099] In some embodiments, the font source 223 may be configured to provide predefined fonts for the different adaptive font generation techniques described above, multiple proxy fonts, a set of labeled fonts, a set of derived reference fonts, or custom fonts. In some examples, the font 215 provided by the font source includes fonts configured to represent specific phonetic features or combinations of phonetic features.

[0100] In one example, the font source 223 may be a font repository, e.g., local or remote storage of fonts. Some of the fonts in the font source 223 may correspond to empirically developed fonts. However, additionally or alternatively, the font source 223 may include a predefined font generation unit (not shown) configured to generate several fonts using a labeling process and cluster analysis based on one or more base fonts or base designs. In one embodiment, this may include using another trained model (e.g., a deep learning model).

[0101] According to some embodiments, a control unit 115 (see Figure 1) of the computing system 100 is configured to provide a control signal 109 to a font processing system 111 (e.g., 200 in Figure 2). In one embodiment, the control signal 109 (e.g., 209 in Figure 2) may be configured to control the generation of the adaptive font 213 according to, for example, user settings and / or preferences. For example, in one embodiment, a specific user interface, such as a configuration menu, may be provided to the user during operation, allowing the user to select the nature of the extensions that the adaptive font 213 should provide. For example, in a preferred embodiment, the specific user interface may include a first user interface element (e.g., a button or switch) for enabling or disabling extensions associated with paralinguistic features of an utterance (e.g., extensions corresponding to emotions detected therein), a second user interface element for enabling or disabling extensions associated with linguistic features of an utterance (e.g., extensions characterizing specific languages ​​and / or accents detected in the utterance), and / or a third user interface element for enabling or disabling extensions associated with phonetic features of an utterance (e.g., extensions representing one or more prosodic features detected in the utterance, e.g., volume, speech rate, etc.). Therefore, users can easily configure the extensions provided.

[0102] (Example method) Figure 10 corresponds to a flowchart representing a method 1000 performed by a computing system such as the computing system 100 in Figure 1, or in particular the processing system 119 (especially the font processing system 111) in Figure 1. This may be performed using one or more software modules, one or more hardware modules, or any combination thereof. For example, this may be at least partially implemented using an extension that can be plugged into any media output device or media output software or application, such as a browser extension. This may be performed on a user computing device, on a remote computing device (e.g., the cloud), or any combination thereof. In particular, all steps described below may correspond to or refer to the functions or actions described in detail above in any of the entities in Figures 1-9. Conversely, the details provided below may also apply to the computing system 100 (especially the font processing systems 111, 200) described above.

[0103] Method 1000 may begin with step 1002, which consists of acquiring audio data 1006 (e.g., audio data 101 or 201). For example, font processing systems 111, 200 receive audio data 1006 from any audio-video content (e.g., movies, video streams) or pure audio content (e.g., audio streams, telephone calls). In one embodiment, audio data 1006 may specifically include voice content (e.g., speech, songs) and may encode a series of speeches from one or more voice sources (e.g., speakers, singers, or characters).

[0104] Furthermore, method 1000 may be followed by step 1004, which consists of obtaining text information 1008 (e.g., text information 105 or 205) (e.g., at least one subtitle or closed caption) that transcribes the utterances contained in the audio data 1006. In some embodiments, this is performed by font processing systems 111, 200, and the text information 1008 may be obtained directly with the audio data 1006 (e.g., as a side channel provided with the audio stream or audio / video stream, e.g., a subtitle / closed caption channel). In another embodiment, the text information 1008 may be obtained using a transcription unit 121 based on the audio data 1006 and then supplied to the font processing systems 111, 200. In one example, the text information 1008 may be specifically associated with each utterance of the audio data 1006, or it may be part of a stream of text information 1008 provided along with the audio data 1006 to transcribe multiple utterances of the audio data 1006. Alternatively, it may be part of a file (e.g., a text file) containing multiple pieces of text information 1008 associated with the corresponding utterances in the audio data 1006.

[0105] Method 1000 may follow step 1010, which involves using a trained model to determine a plurality of feature parameters 1012 (e.g., 211) associated with a plurality of speech features for an utterance. In some embodiments, step 1010 corresponds to a function specifically performed by the speech analysis unit 217 of the font processing systems 111, 200. As a result of step 1010, a plurality of feature parameters 1012 characterizing the utterance currently being processed are provided.

[0106] Method 1000 may also be followed by step 1014, which consists of determining an adaptive font 213 based on a plurality of feature parameters 1012, where at least one embodiment of the adaptive font is adapted based on at least one of the plurality of feature parameters 1012. This step may be performed by the adaptive font generation unit 219 of the font processing systems 111, 200, specifically according to, for example, variable font generation techniques (see unit 219a), discrete value generation techniques (see unit 219b), custom font modification techniques (see unit 219c), or AI-based generation techniques (see unit 219d), as described above. In some examples, step 1014 may specifically include the use of remote computing or storage resources, such as a cloud server, to determine the adaptive font 213.

[0107] Finally, method 1000 may be followed by step 1018, which consists of formatting at least a portion of the text information 1008 using adaptive font 213 for display to the user. The result of this step is formatted text information 107, 207 suitable for display to the user and including enhancements achieved by adaptive font 213. See the details above, in particular the details concerning the font application unit 221 of the font processing systems 111, 200. In some cases, the entire text information 1008 is formatted using adaptive font 213. In some other cases, as described above, at least a portion of it is formatted using a modified adaptive font based on a set of associated local feature parameters in order to optimally render local paralinguistic or phonetic deviations in utterances, such as emphasis on words or terms.

[0108] In a particularly preferred embodiment, steps 1004, 1010, 1012, 1014, and 1018 may be repeated as many times as necessary, for example, as long as new utterances are detected and / or obtained from the audio data 1006. In some embodiments, the process may even return to step 1002 to obtain new / further audio data 1006. Thus, the sequence of utterances contained in the audio data 1006 may be processed, for example at runtime, to provide a corresponding set of formatted text information 107, 207 that transcribes the corresponding sequence of utterances of the audio content.

[0109] In one embodiment, method 1000 may further include step 1020 of displaying formatted text information 107, 207 to the user on a display (e.g., output device 113). For example, it may be displayed as subtitles on the output display 113.

[0110] Furthermore, in yet another embodiment, method 1000 may further include the step of receiving control signals 109, 209 for controlling the determination of the adaptive font 213 based on user settings and / or preferences. For example, as described above in Figure 1 or Figure 2, this step may include receiving a first control signal (e.g., using a first user interface element) for enabling or disabling extensions related to paralinguistic features of speech, a second control signal (e.g., using a second user interface element) for enabling or disabling extensions related to linguistic features of speech, and / or a third control signal (e.g., using a third user interface element) for enabling or disabling extensions related to phonetic features of speech, via a user interface displayed to the user. In a preferred embodiment, this user interface may be part of a user interface for controlling subtitle settings (e.g., language settings), and may also be implemented using specific extensions, such as browser extensions.

[0111] (Other use cases for adaptive font development) In addition to the embodiments described above, a user interface is envisioned that provides a display of fonts representing calculated effects or language categories to the viewer. In some examples, this provision is based on one or more manual assignments or settings provided by chat users, publishers, authors in chat rooms, and / or for digital and analog (print) media. In one embodiment, the user interface may include additional functionality to configure several visual aspects, such as color or animation.

[0112] In some examples, this user interface is configured to define font assignments or specific font associations for one or more individuals, organizations, companies (as part of their corporate identity), and / or products, text messages, designs, or campaigns. Such definitions can be performed based on manual assignments or using algorithmic cluster analysis. The user interface aims to assign one or more intended effects (e.g., intended paralinguistic effects such as association with joy or association with technological advancement) based on the expected effect the font has on the viewer. In one embodiment, one or more intended effects of a design can be determined based on text input (e.g., a product description or website) by algorithmic analysis and several linguistic models.

[0113] (Use cases for adaptive font generation) The following discloses further representative examples of adaptive font generation and usage.

[0114] For example, some embodiments of the present disclosure can be defined by the following examples. Furthermore, embodiments or aspects of the present disclosure can be combined with the following examples in any combination.

[0115] (Example 1) A method for visually displaying speech using fonts against an optically displayed background or film, characterized in that at least one voice feature or phonetic feature of the voice speaking the speech is visually displayed using a graphically displayed font, using the respective optical presentations of the fonts associated with each voice feature or phonetic feature.

[0116] (Example 2) A method according to Example 1, characterized in that multiple or different voice characteristics or phonetic characteristics are displayed by different typography for a font or by typographic presentation of a font.

[0117] (Example 3) A method according to Example 1 or 2, characterized in that the complete sentence is displayed by a font.

[0118] (Example 4) A method relating to any one of the above examples, characterized in that different presentation fonts are used for the visual representation of different national languages, accents, social dialects, or dialects.

[0119] (Example 5) A method according to any one of the above examples, characterized in that different presentation fonts are sequentially followed by each other for a font used for the visual representation of a time-changing voice or speech feature.

[0120] (Example 6) A method according to any one of the above examples, characterized in that a voice volume profile is represented by a font, and at least one character presented in a particular manner is associated with a particular volume or volume range.

[0121] (Example 7) A method according to any one of the above examples, characterized in that the speech rate of a voice is represented by a font relative to a font, and at least one character presented in a particular way is associated with a particular speech rate.

[0122] (Example 8) A method relating to any one of the above examples, characterized in that spatial auditory perception is simulated by the orientation of the font by associating a font with each image edge on which each stereophonic sound source radiates an audio signal into space.

[0123] (Example 9) A method relating to any one of the above examples, characterized in that paralinguistic features of a voice related to the emotional expression of a speaker are represented by a font, and at least one specific presentation of a character is associated with a specific emotional expression.

[0124] (Example 10) An arrangement in which the method of any one of the above examples can be performed or is performed, the arrangement comprising a medium, wherein an algorithm performed by a device displays a method for converting an audio signal of the medium, radiated into space in parallel with at least one background that is optically displayed by the medium, into optically displayed characters, wherein at least one acoustic or physical feature of the audio signal is optically displayed on the medium by an optical structure of one or more characters associated with each feature.

[0125] A preferred embodiment relates to the method described in the preamble of Example 1 and an arrangement for carrying out the method.

[0126] Further details are provided below.

[0127] Subtitles are provided for films to convey their content to people with hearing impairments or those who cannot understand the spoken language. Currently, neutral fonts are used as much as possible to visually represent the speech. Font groups with fewer distinctive features, such as grotesque or monospace fonts, can be used as fonts associated with German-speaking regions.

[0128] However, known subtitles only convey the linguistic content and cannot express the audio linguistic context of a film, the mood and linguistic characteristics of the actors, and especially their emotions.

[0129] Therefore, preferred embodiments are based on the objective of visually displaying information transmitted by an acoustic event or audio signal of speech.

[0130] A preferred embodiment achieves the above-mentioned objectives by the features of Example 1.

[0131] In a preferred embodiment, since spoken language is often displayed only by neutral and featureless fonts, it was first recognized that fonts, especially fonts used for subtitles, have historically been a very one-dimensional experience.

[0132] It has become increasingly recognized that specific font representations are necessary to give diversity and proximity to purely optically displayed images or films.

[0133] Specifically, it was recognized that, against this background, at least one vocal characteristic of a language-speaking voice needs to be visually represented in a graphically displayed font using an optical presentation of the font associated with each vocal characteristic.

[0134] Each presentation can visually convey not only basic emotions such as sadness, fear, anger, and joy, but also love and rejection.

[0135] According to this invention, the font used is adapted to the sound of speech. Therefore, people with hearing impairments can participate more in language and culture, and everyone can have a deeper reading experience.

[0136] The described method allows for the visual representation of acoustic events in spoken language. Various linguistic aspects can be visually translated.

[0137] Multiple or different vocal or phonetic features can be represented by different typography for a font or by the typographic presentation of a font. For example, it is possible to represent hard sounds with hard-looking characters and soft sounds with soft-looking characters. Hard language sections may be represented, for example, by angular or angular characters, or by characters with sharp shapes, while soft language sections are likely to be represented by rounded or smooth characters. The presentation of characters conveys the respective characteristics of spoken language.

[0138] The entire sentence can be displayed using a font. In contrast to the visually limited display of only individual words, the entire sentence can more fully represent the emotional or acoustic progression of the voice, maintaining the flow of reading. Furthermore, if different fonts are used for different voices, the voices of different people can be more clearly associated with the person speaking.

[0139] Different fonts can be used to visually represent different languages ​​or dialects. For example, to represent a language, typography can be used that evokes the viewer's image of that country, as the typography integrates symbols and fonts characteristic of that country into the characters. Social dialects, individual language errors, or accents can also be visually represented through the typography of each font.

[0140] For the visual representation of time-changing vocal or phonetic features, different font presentations can be sequentially followed by each other. Specifically, the presentation of characters can change in the course of a sentence as these characters are displayed sequentially over time. For example, a spoken sentence can begin with the first presented character and end with the second or further presented characters. Thus, it is possible to represent that a voice initially speaks without accent and then with accent. To represent a spoken sentence, characters can be superimposed sequentially over time, and the presentation of the sequentially superimposed characters changes. Thus, for example, vocal features occurring at that moment, such as changes in volume, can be visually represented.

[0141] The volume profile of a voice can be represented by a font, where at least one character, presented in a specific way, is associated with a particular volume or volume range. Thus, whether a voice is speaking loudly or softly can be indicated by a specific character. Specifically, for example, the thickness of the line of a character used to represent a louder volume can be increased.

[0142] The timbre of a voice can be represented by the font, with at least one character presented in a specific way associated with a particular frequency range. Therefore, whether a voice is speaking darkly or brightly can be indicated by a specific character.

[0143] Fonts can blend smoothly with each other. Interpolation of different fonts, that is, a smooth transition from one font to another, can be done in the context of several auditory features occurring simultaneously within a typographic display. Therefore, for example, the quiet anger in Spanish can be visually represented through the interpolation of different fonts. For instance, spoken sentences can visually reproduce their volume, emotion, and linguistic aspects.

[0144] The rate of speech can be represented by the font used, with at least one character, presented in a specific way, associated with a particular rate of speech. Thus, certain characters can express whether the voice is hurried, subdued, or cold.

[0145] By associating fonts with each image edge of the screen where each stereophonic sound source radiates an audio signal into space, spatial hearing can be simulated through font orientation. Therefore, for example, if the left speaker is radiating an audio signal, the font can be left-aligned; if the right speaker is radiating an audio signal, the font can be right-aligned. In this way, the spatial distribution of sound can be visually displayed to people with hearing impairments.

[0146] Paralinguistic features of speech related to the emotional expression of a speaker can be represented by fonts, where at least one specific representation of a character is associated with a particular emotional expression. Basic emotions, particularly sadness, fear, anger, joy, affection, or rejection, as phonetic or phonological features, can be represented by fonts and communicated optically. For example, graphical symbols associated with basic emotions can be integrated into fonts. Such symbols could be, for example, tears associated with the basic emotion of sadness.

[0147] In this explanation, a specific font representation refers to the specific optical appearance of characters, but is understood to also include, in some cases, emojis, punctuation marks, and special characters used.

[0148] A configuration that includes a medium can be used to perform the types of methods described herein.

[0149] This configuration may include a medium that displays an algorithm performed by the device that converts an audio signal of a medium, emitted into space in parallel with at least one background that is optically displayed by the medium, into optically displayed characters. At least one acoustic or physical feature of the audio signal can be optically displayed on the medium by optical structures of one or more characters associated with each feature.

[0150] Specifically, the medium can consist of a television, laptop, computer, bus, train or in-train screen, or mobile phone. Therefore, the auditory content of a film or image can be described using fonts that convey the auditory characteristics of what is being described.

[0151] Further possibilities and aspects for or in which the methods described herein are carried out are optionally disclosed below.

[0152] A method is disclosed for using representative screen-optimized font sections to enable barrier-free mediation of spoken language culture through typography. With the help of this method, one-dimensional media fonts are enhanced for diversity and accessibility, and audio information previously inaccessible to the hearing impaired is transcribed. Its use in subtitles results in a deeper cinematic experience.

[0153] A method for visualizing auditory attributes of a speech process is disclosed through the automated use of representative font sections from different microtypographic designs.

[0154] This method simplifies the participation of people with hearing impairments in language culture, for example, by using depicted fonts, and provides information about the phonetic characteristics of spoken language in audiovisual media subtitles.

[0155] A collection of auditory stimuli is linked to visual stimuli according to a defined pattern, associated with microtypography, and formed in the recipient's ideas, by linking individual auditory signals through a) symbolism (meaning), b) physical links, c) specific associations (identification of objects / ideas), d) intermode analogies (correlation of features), or e) observation of general synesthetic perceptual phenomena.

[0156] Linguistic aspects to consider, to the advantage of them, include a) certain phonetic features of different national languages, accents, dialects, or social dialects; b) certain and transient individual phonetic features (e.g., speech rate, volume, pitch, emphasis); and c) transient emotional expressions (paralinguistics).

[0157] The linguistic units that utilize this method are primarily sentences, and in individual cases, individual words, in order to maintain a consistent design and readability.

[0158] Language sections with a high consonant density, particularly those containing fricatives and plosives, and where these consonants are strongly emphasized, appear softer and are represented with a more angular, pointed shape than language sections with many vowels and where these vowels are strongly emphasized. These are displayed in a rounded, curvilinear manner.

[0159] An increase in volume can be corresponded to an increase in font thickness. An increase in speech speed can be corresponded to a decrease in font width. An increase in pitch can be corresponded to an increase in upper length.

[0160] Passive emotions are displayed in a font tilted in the opposite direction of reading, while active emotions are displayed in a font tilted in the direction of reading. Positive emotions are displayed in an upward-pointing font, and negative emotions are displayed in a downward-pointing font.

[0161] Direction refers to the orientation of individual letter elements, the guidance of corresponding lines, and the use of partially corresponding serifs. A typical spring angle of 45° represents harmony, while deviation angles that increase with deviation represent disharmony.

[0162] A contrast in line thickness of uniform thickness represents balance. A contrast in line thickness of high thickness represents imbalance. Letters usually imitate facial expressions and gestures associated with emotions. Symbols such as tears representing sadness are also incorporated into letters. The static characteristics of letters reflect the speaker's emotional stability. Disharmony is expressed by a break in visual conventions. Letter spacing corresponds to sound density.

[0163] Each linguistic context, such as the phonetic peculiarities of individual characters in terms of accent and linguistic errors, like a rolled R or a silent h, is reflected in the typographically translated form of each character. Each glyph evokes the font or typeface preferred in the native language, such as the Cyrillic alphabet. Sounds that deviate from the standard are represented by the glyph's end pointing in the corresponding direction.

[0164] Spatial auditory simulation is achieved by the orientation of fonts or text at the edges of the image of the corresponding stereo sound source.

[0165] One use case involves a screen displaying an image or sequence of images as a background, where the utterances of the image or sequence of images are visually displayed, and the utterances are emitted into space from a speaker via audio signals.

[0166] This use case illustrates a configuration in which the types of methods described below are performed.

[0167] This configuration comprises a medium, the medium displaying a method by which an algorithm performed by the device converts at least one background, specifically an image, that is optically displayed by the medium, and an audio signal of the medium emitted into space in parallel with the image, into optically displayed characters.

[0168] At least one acoustic or physical feature of an audio signal is optically displayed on a medium by an optical structure of one or more characters associated with each feature.

[0169] The medium referred to here is specifically television, on which the following types of methods are performed.

[0170] A television has a screen on which movie scenes or images are displayed, and on that screen, speech is spoken, which is visually displayed by the font of subtitles.

[0171] The audio signal, that is, the speaker's voice, is displayed in a general form and emitted as an acoustic signal by a speaker into the space where the medium is placed. The acoustic signal is then visualized visually in the following way:

[0172] In this method, a speech utterance is visually represented in relation to an image by visually representing at least one phonetic or phonological feature of the speech utterance using a graphically displayed font. Specifically, the respective optical representations of the fonts associated with each phonetic or phonological feature are used.

[0173] Multiple or different phonetic or phonological features are represented by different typography or typographic representations of fonts for the subtitles. The entire sentence is represented by the font.

[0174] The audio volume profile is represented by the font, and at least one character displayed in a specific way is associated with a particular volume or volume range.

[0175] The fonts specified by the algorithm for the words "thicker," "thinner," "broader," "narrower," "anxious," and "raging" represent the specific phonetic characteristics of the speech during the utterance of each word.

[0176] The speech rate is indicated by the font used for the subtitles, with at least one character displayed in a specific way associated with a particular speech rate.

[0177] The medium described herein illustrates how device 6 converts the audio signal of the medium, i.e., sound, emitted into a space linked to an optically displayed background, i.e., image, on the medium, into optically displayed characters, i.e., subtitles, using an algorithm.

[0178] At least one acoustic or physical feature of an audio signal, i.e., a linguistic or phonetic feature, is optically displayed on a medium by an optical structure associated with each feature, i.e., an optical representation of one or more characters.

[0179] In this regard, the method is carried out by a configuration in which the utterance is visually displayed relative to an image by visually representing at least one phonetic or phonological feature of the speech being spoken using a graphically displayed font. Specifically, the respective optical representations of the fonts associated with each phonetic or phonological feature are used.

[0180] (An exemplary computing system) The embodiments, operations, and behaviors of the subject matter described herein can be implemented in digital electronic circuits, tangibly embodied computer software or firmware, computer hardware including the structures disclosed herein and their structural equivalents, or one or more combinations thereof. The embodiments of the subject matter described herein can be implemented as modules of one or more computer program instructions encoded in a computer program carrier to control one or more computer programs, for example, execution by a data processing device or the operation of a data processing device. For example, the computer program carrier may include one or more computer-readable storage media on which the instructions are encoded or stored. The carrier may be a tangible, non-transient computer-readable medium such as a magnetic disk, magneto-optical disk, or optical disk, solid-state drive, random-access memory (RAM), read-only memory (ROM), or other types of media. Alternatively or additionally, the carrier may be an artificially generated propagating signal, for example, a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiving device for execution by a data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage substrate, a random or serial-access memory device, or one or more combinations thereof, or a part thereof. Computer storage media are not propagating signals.

[0181] Computer programs, also called or written as programs, software, software applications, apps, modules, software modules, engines, scripts, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and can be deployed as standalone programs or in any form, including modules, components, engines, subroutines, or other units suitable for execution in a computing environment which may include one or more computers interconnected by a data communication network in one or more locations.

[0182] Computer programs can, but do not necessarily, correspond to files in a file system. They can be part of a file that holds other programs or data, such as one or more scripts stored in a markup language document, a single file dedicated to the program, or multiple collaborative files, such as one or more modules, subprograms, or files containing parts of code.

[0183] Processors for executing computer programs include, for example, both general-purpose and dedicated microprocessors, and any one or more processors in any type of digital computer. Generally, a processor receives instructions for the computer program to be executed, as well as data from non-temporary computer-readable media coupled to the processor.

[0184] The term "data processing device" encompasses all kinds of devices, machines, and equipment for processing data, including, for example, programmable processors, computers, or multiple processors or computers. A data processing device may include dedicated logic circuits such as FPGAs (Field-Programmable Gate Arrays), ASICs (Application-Specific Integrated Circuits), or GPUs (Graphics Processing Units). In addition to hardware, a device may also include code that constitutes the execution environment for computer programs, such as processor firmware, protocol stacks, database management systems, operating systems, or one or more combinations thereof.

[0185] The processes and logic flows described herein can be executed by one or more computers or processors running one or more computer programs that perform operations by manipulating input data to produce outputs. The processes and logic flows can also be executed by dedicated logic circuits, such as FPGAs, ASICs, or GPUs, or by a combination of dedicated logic circuits and one or more programmed computers.

[0186] A computer suitable for running computer programs can be based on a general-purpose or dedicated microprocessor, or both, or other types of central processing units. Generally, the central processing unit receives instructions and data from read-only memory or random-access memory, or both. The elements of a computer can include a central processing unit for executing instructions and one or more memory devices for storing instructions and data. The central processing unit and memory can be complemented by or incorporated into dedicated logic circuits.

[0187] Generally, a computer also includes one or more storage devices, or is operable to receive data from or transfer data to one or more storage devices. Storage devices may be, for example, magnetic disks, magneto-optical disks, or optical disks, solid-state drives, or other types of non-temporary computer-readable media. However, a computer is not required to have such devices. Therefore, a computer can be coupled to one or more storage devices, such as one or more local and / or remote memories. For example, a computer may include one or more local memories that are integral components of the computer, or a computer may be coupled to one or more remote memories located in a cloud network. Furthermore, a computer can be embedded in another device, such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or portable storage device, such as a Universal Serial Bus (USB) flash drive.

[0188] Components can be “coupled” to one another by being connected to each other in a communicative manner (e.g., electrically or optically) directly or through one or more intermediate components. Components can also be “coupled” to one another if one of them is integrated into the other. For example, a memory component integrated into a processor (e.g., an L2 cache component) is “coupled” to the processor.

[0189] To provide user interaction, embodiments of the subject matter described herein may be implemented on or configured to communicate with a computer having a display device for displaying information to the user, such as an LCD (liquid crystal display) monitor, and an input device, such as a keyboard and a pointing device, such as a mouse, trackball, or touchpad, to which the user can provide input to the computer. Other types of devices may also be used to provide user interaction. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or haptic feedback, and input from the user may be received in any form, including acoustic, voice, or haptic input. Furthermore, the computer may interact with the user by sending documents to and receiving documents from the user's device. For example, it may interact by sending a web page to a web browser on the user's device in response to a request received from a web browser, or by interacting with an application running on the user's device, such as a smartphone or electronic tablet. The computer may also interact with the user by sending text messages or other forms of messages to a personal device, such as a smartphone running a messaging application, and receiving response messages from the user.

[0190] In this specification, the term “configured to” is used in relation to components of systems, devices, and computer programs. One or more computer systems are configured to perform a particular operation or action to mean that the system has installed software, firmware, hardware, or a combination thereof that causes the system to perform the operation or action when in operation. One or more computer programs are configured to perform a particular operation or action to mean that one or more programs, when executed by a data processing device, contain instructions that cause the device to perform the operation or action. A dedicated logic circuit is configured to perform a particular operation or action to mean that the circuit has electronic logic that performs the operation or action.

[0191] This specification includes details of many specific embodiments, but these should not be interpreted as limitations on the claimed scope as defined by the claims themselves, but rather as descriptions of features that may be specific to particular embodiments. Certain features described in this specification in the context of separate embodiments may also be realized in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be realized separately or in any suitable partial combination in multiple embodiments. Furthermore, features are described above as acting in a particular combination, and may initially be claimed as such, but one or more features from the claimed combination may, in some cases, be removed from the combination, and the claims may be directed to partial combinations or variations of partial combinations.

[0192] Similarly, although the operations are depicted in the drawings and described in a specific order in the claims, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all illustrated operations be performed, in order to achieve the desired result. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems can generally be integrated into a single software product or packaged into multiple software products.

[0193] While several embodiments have been described in detail, it should be understood that the embodiments of this disclosure can take many forms. In particular, the claimed subject matter may be carried out or implemented in ways different from the examples described, and the described features and characteristics may be carried out or implemented in any combination. For example, the process depicted in the accompanying figures does not necessarily require the specific order or sequential order shown to achieve the desired result. In some cases, multitasking and parallel processing may be advantageous. The embodiments shown herein are intended to be illustrative rather than limiting to the invention as defined by the claims.

[0194] (An exemplary viewpoint) Embodiments of this disclosure can be defined by any combination of one or more of the following aspects:

[0195] (Perspective 1) A method performed by a computer, comprising: acquiring audio data; acquiring text information by transcribing utterances contained in the audio data; determining a plurality of feature parameters related to a plurality of speech features for an utterance using a trained model; determining an adaptive font based on the plurality of feature parameters, wherein at least one aspect of the adaptive font is adapted based on at least one of the plurality of feature parameters; and formatting at least a portion of the text information for display to a user using the adaptive font.

[0196] (Perspective 2) A method relating to Perspective 1, wherein the utterance is characterized by multiple feature parameters.

[0197] (Perspective 3) A method according to Perspective 1 or 2, wherein the plurality of feature parameters include at least one of one paralinguistic parameters relating to one or more paralinguistic features, one or more phonetic parameters relating to one or more phonetic features, and / or one or more phonetic parameters relating to one or more phonetic characteristics.

[0198] (Perspective 4) A method relating to Perspective 3, wherein one or more paralinguistic parameters include one or more parameters related to one or more candidate emotions, and each represents an estimate indicating whether an utterance exhibits the respective candidate emotion.

[0199] (Perspective 5) A method according to Perspective 4, wherein one or more candidate emotions include one or more of anger, joy, fear, and / or sadness.

[0200] (Perspective 6) A method relating to a linguistic characteristic, wherein one or more phonetic parameters include one or more parameters indicating the detected language of the utterance, one or more parameters indicating the detected accent of the utterance, one or more confidence levels for one or more candidate languages, each indicating the confidence that the utterance is in the respective candidate language, and / or one or more confidence levels relating to one or more candidate accents, each indicating the confidence that the utterance is in the respective candidate accent.

[0201] (Perspective 7) A method according to any one of Perspectives 3 to 6, wherein one or more speech parameters relating to one or more speech characteristics include one or more parameters indicating the volume level of an utterance, a parameter indicating the speech rate of an utterance, and / or a parameter indicating the pitch of an utterance.

[0202] (Perspective 8) A method according to any one of the above perspectives, wherein multiple feature parameters are stored as multiple values ​​of a vector, preferably in an array or matrix.

[0203] (Perspective 9) A method relating to any one of the preceding perspectives, wherein the trained model is a speech emotion recognition model, and comprises at least one machine learning model, a deep learning model, a large language model, and / or a natural language processing (NLP) model.

[0204] (Perspective 10) A method relating to any one of the above perspectives, wherein determining a plurality of feature parameters for an utterance using a trained model comprises extracting one or more audio features from a portion of audio data relating to an utterance, inputting the portion of audio data and the one or more audio features into a trained model, and obtaining a plurality of feature parameters from the trained model.

[0205] (Perspective 11) A method relating to any one of the above perspectives, further comprising identifying one or more important speech features for an utterance based on a plurality of feature parameters, wherein one or more important speech features and one or more determined feature parameters of the plurality of feature parameters are representative of the utterance, and determining an adaptive font based on the plurality of feature parameters comprises determining an adaptive font based on one or more determined feature parameters of one or more important speech features.

[0206] (Perspective 12) A method according to Perspective 11, wherein one or more important phonetic features are associated with one or more predetermined types of adaptations, and determining an adaptation font based on one or more determined feature parameters of one or more important phonetic features includes applying one or more predetermined types of adaptations based on one or more determined feature parameters of one or more important phonetic features.

[0207] (Perspective 13) A method relating to Perspective 12, further comprising determining an application coefficient for a predetermined type of adaptation to one or more important speech features based on a determined feature parameter associated with the important speech feature, and applying one or more predetermined types of adaptations based on one or more determined feature parameters of one or more important speech features, comprising applying one or more predetermined types of adaptations to a predefined font at a ratio correlated with the application coefficient of one or more predetermined types of adaptations in order to generate an adaptive font.

[0208] (Perspective 14) A method relating to Perspective 13, wherein for each of one or more important speech features, the application coefficient of a predetermined type of adaptation correlates with the determined feature parameter of the important speech feature.

[0209] (Perspective 15) A method according to Perspective 13 or 14, wherein for each important speech feature of one or more important speech features, the application coefficient of each predetermined type of adaptation lies on a scale having a minimum and a maximum value, the minimum value representing a first degree of application of the predetermined type of adaptation and the maximum value representing a second degree of application of the predetermined type of adaptation.

[0210] (Perspective 16) A method according to any of Perspectives 13 to 15, wherein the predefined font is a variable font in which one or more aspects can be progressively adapted.

[0211] (Perspective 17) A method according to any of Perspectives 13 to 16, wherein one or more important phonetic features include a plurality of important phonetic features having a plurality of predetermined types of adaptations associated with them, and applying one or more predetermined types of adaptations to a predefined font at a rate correlated with the application coefficient of one or more predetermined types of adaptations, the method comprising applying each of the plurality of predetermined types of adaptations to the predefined font at a rate correlated with the application coefficient of each of the predetermined types of adaptations.

[0212] (Perspective 18) A method according to any of Perspectives 13 to 17, wherein one or more important phonetic features include a first phonetic feature from a paralinguistic feature category and are associated with a first type of adaptation, a first application coefficient is determined based on determined feature parameters associated with the first phonetic feature in a plurality of feature parameters, and applying one or more predetermined types of adaptations to a predefined font at a ratio correlated with the application coefficient of one or more predetermined types of adaptations is a method comprising applying a first type of adaptation to a predefined font at a ratio correlated with the first application coefficient.

[0213] (Perspective 19) A method relating to Perspective 18, wherein the first phonetic feature is activity, and the first type of adaptation is to tilt the orientation of one or more characters of a predefined font in a direction corresponding to the reading direction.

[0214] (Perspective 20) A method relating to Perspective 18, wherein the first phonetic feature is a candidate emotion of the fear type, and the adaptation of the first type is to tilt the orientation of one or more characters of a predefined font in the direction opposite to the reading direction.

[0215] (Perspective 21) A method according to any of Perspectives 1 to 10, wherein determining an adaptive font based on a plurality of feature parameters includes selecting one proxy font as the adaptive font from a plurality of proxy fonts based on a plurality of feature parameters.

[0216] (Perspective 22) A method relating to Perspective 21, wherein a plurality of proxy fonts correspond to a plurality of predetermined fonts, and each of the plurality of proxy fonts is associated with a respective predetermined set of values ​​for the characteristic parameters of a plurality of speech features.

[0217] (Perspective 23) A method relating to Perspective 22, wherein selecting one proxy font as an adaptive font from a plurality of proxy fonts based on a plurality of feature parameters includes selecting from a plurality of proxy fonts the proxy font whose set of predetermined values ​​is closest to the plurality of feature parameters.

[0218] (Perspective 24) A method according to Perspective 21, wherein a plurality of proxy fonts correspond to a plurality of predetermined fonts generated for at least a range of discrete values ​​for each of one or more feature parameters of one or more important speech features, each proxy font is associated with a particular combination of discrete values ​​that form its respective set of proxy coordinates, and selecting one proxy font as an adaptive font from the plurality of proxy fonts based on a plurality of feature parameters includes selecting one proxy font from a plurality of proxy fonts whose set of proxy coordinates is closest to one or more determined feature parameters of one or more important speech features.

[0219] (Perspective 25) A method according to any of Perspectives 1 to 10, wherein determining an adaptive font based on multiple feature parameters further comprises using a trained model to generate an adaptive font based on multiple feature parameters.

[0220] (Perspective 26) A method of the method of Perspective 25, wherein a further trained model is a model trained on a set of labeled fonts, each representing a specific speech feature, in order to determine reference associations between a set of combinations of values ​​of a set of feature parameters and a set of derived reference fonts, and a method of generating an adaptive font based on a set of feature parameters using the further trained model, comprising generating an adaptive font based on a set of feature parameters and reference associations.

[0221] (Perspective 27) A method of the method described in Perspective 26, wherein generating an adaptive font based on a plurality of feature parameters and reference associations comprises applying to one or more fonts from a derived set of reference fonts one or more adaptations of one or more microtypographic perspectives or one or more microtypographic elements, one or more adaptations of one or more macrotypographic perspectives or one or more macrotypographic elements, one or more adaptations of one or more display characteristics or one or more display elements, and one or more adaptations of one or more dynamic characteristics.

[0222] (Perspective 28) A method according to Perspective 26 or 27, wherein generating an adaptive font based on a plurality of feature parameters and reference associations comprises generating an adaptive font based on a plurality of feature parameters and reference associations using either a stable diffusion process or a character skeleton mapping process.

[0223] (Perspective 29) A method relating to Perspective 11, wherein determining an adaptive font based on one or more determined feature parameters of one or more important phonetic features comprises applying one or more custom type adaptations to a custom font based on one or more determined feature parameters of one or more important phonetic features.

[0224] (Perspective 30) A method relating to Perspective 29, wherein the adaptation of one or more custom types is defined according to one or more custom rules for a custom font.

[0225] (Perspective 31) A method according to any of Perspectives 13 to 30, wherein a predefined font, a set of proxy fonts, a set of labeled fonts, a set of derived reference fonts, or a custom font is obtained from a font source.

[0226] (Perspective 32) A method relating to Perspective 31, wherein some fonts in a font source are generated using a labeling process and cluster analysis based on one or more basic fonts or one or more basic designs.

[0227] (Perspective 33. A method relating to any of the preceding perspectives, wherein the utterance is a unit of speech.)

[0228] (Perspective 34) A method of Perspective 33, wherein a plurality of feature parameters associated with a plurality of speech features determined for an utterance correspond to a first plurality of feature parameters, and the method further comprises determining one or more subunits of an utterance within an utterance unit having a second plurality of feature parameters each which substantially deviates from the first plurality of feature parameters with respect to at least one speech feature, and determining a modified adaptive font for one or more subunits of an utterance based on a quantified deviation between each of the second plurality of feature parameters and the first plurality of feature parameters, and formatting at least a portion of the text information based on the adaptive font for display to a user, comprising formatting one or more portions of the text information corresponding to one or more subunits of an utterance using the modified adaptive font.

[0229] (Perspective 35) A method of any of the preceding perspectives, wherein determining an adapted font based on multiple feature parameters includes one or more of the following: adapting the typeface of a predefined font to bold when one or more feature parameters from the multiple feature parameters indicate a high volume level of speech; adapting the typeface of a predefined font to light when one or more feature parameters from the multiple feature parameters indicate a low volume level of speech; reducing the character spacing and / or character width of a predefined font when one or more feature parameters from the multiple feature parameters indicate a high speech rate; increasing the character spacing and / or character width of a predefined font when one or more feature parameters from the multiple feature parameters indicate a low speech rate; A method comprising: tilting the orientation of a predefined font in a direction corresponding to the reading direction when one or more feature parameters represent a prominent presence of activity in an utterance; tilting the orientation of a predefined font in a direction opposite to the reading direction when one or more feature parameters from a plurality of feature parameters represent a prominent presence of passivity in an utterance; adapting the vertical orientation of one or more characters of a predefined font by turning the ends of one or more characters upward when one or more feature parameters from a plurality of feature parameters represent a prominent presence of joy or positive emotion in an utterance; and adapting the vertical orientation of one or more characters of a predefined font by turning the ends of one or more characters downward when one or more feature parameters from a plurality of feature parameters represent a prominent presence of sadness or negative emotion in an utterance.

[0230] (Perspective 36) A method relating to any of the preceding perspectives, wherein the method is for conveying the oral characteristics of speech in audio data to a person who is deaf or hard of hearing.

[0231] (Perspective 37) A method relating to any of the preceding perspectives, further comprising determining audio data from one or more of the following: video, audio files, audio broadcasts, phone calls, live audio events or virtual audio events, or video conferences.

[0232] (Perspective 38) A method relating to any of the preceding perspectives, further comprising determining audio data from video and providing at least a portion of formatted text information as subtitles for the video.

[0233] (Perspective 39) One or more computing devices comprising one or more processors configured to perform a method described in any of the preceding perspectives.

[0234] (Perspective 40) One or more computer-readable storage media that, when executed by one or more processors, stores instructions causing the one or more processors to perform the method described in any of Perspectives 1 to 38.

[0235] While several embodiments have been described in detail, it should be understood that the aspects of this disclosure can take many forms. In particular, the claimed subject matter may be carried out or implemented in ways different from those described, and the described features and characteristics may be carried out or implemented in any combination. The embodiments shown herein are intended to be illustrative, not limiting, to the invention as defined by the claims. [Explanation of Symbols]

[0236] [Figure 1] 117: Audio / Audio-Video Processing Unit 111: Font Processing System 115: Control Unit 113: Output device [Figure 3] ANALYSIS:Analysis TIMESTAMP: Timestamp EMOTION HATE: Disgust, JOY: Joy FEAR: Fear GRIEF: Sadness VOICE: Voice VOLUME: Volume SPEED: Speed PITCH: Pitch LANGUAGE: Language ENGLISH: English ACCENT: accent BRITISH: British style [Figure 4] The font can become bolder and thinner, wider and narrower, and look anxious and angry. [Figure 5] OUTPUT: Output SENTENCE: sentence TYPOGRAPHY: Typography WEIGHT: weight SLANT: Slope WIDTH: width AXIS: axis CURVE: Curve OPACITY: Opacity SPACING: Interval SIZU: Size DESIGN: Design OUTLINE: Outline COLOUR: Color GRAIN: Grain size DISTORSION: Distortion MOVEMENT: movement [Figure 6] PASSIVE: Passive NEUTRAL: Neutral ACTIVE: Active WHISPER: whisper SCREAM: Scream DISHARMONY:disharmony HARMONY:Harmony ACTIVITY: activity level VOLUME: Volume HARMONY:Harmony RESULT:Result [Figure 7] ACTIVITY: Activities VOLUME: Volume HARMONY:Harmony EXAMPLE: Example [Figure 8] VOLUME: Volume ACTIVITY: Activities ANGER: anger ANALYSIS:Analysis [Figure 10] 1002: Retrieve audio data 1004: Retrieve text information 1010: Determine feature parameters 1012: Feature Parameters 1014: Determine the appropriate font. 1018: Format text information 1020: Display formatted text information

Claims

1. A computer implementation method, The process of acquiring audio data, The process of obtaining text information by transcribing the utterances contained in the aforementioned audio data, The process involves using a trained model to determine multiple feature parameters associated with multiple speech features for the utterance, A method comprising the steps of: determining an adaptive font based on a plurality of feature parameters, wherein at least one embodiment of the adaptive font is adapted based on at least one of the plurality of feature parameters; and formatting at least a portion of the text information using the adaptive font for display to a user.

2. The method according to claim 1, wherein the utterance is characterized by the plurality of feature parameters.

3. The method according to claim 1 or 2, wherein the plurality of feature parameters include at least one of the following: one or more paralinguistic parameters associated with one or more paralinguistic features, one or more phonetic parameters associated with one or more phonetic features, and / or one or more voice quality parameters associated with one or more voice quality.

4. The method according to any one of the preceding claims, wherein the plurality of feature parameters are stored as a plurality of values ​​of a vector, preferably in an array or matrix.

5. The method according to any one of the preceding claims, wherein the trained model is a speech emotion recognition model and comprises at least one of a machine learning model, a deep learning model, a large language model, and / or a natural language processing (NLP) model.

6. The method according to any one of the preceding claims, further comprising the step of identifying one or more important voice features for an utterance based on the plurality of feature parameters, wherein the one or more important voice features and the one or more determined feature parameters in the plurality of feature parameters are representative of the utterance, and the step of determining the adaptive font based on the plurality of feature parameters comprises the step of determining the adaptive font based on the one or more determined feature parameters of the one or more important voice features.

7. The method according to claim 6, wherein the one or more important voice features are associated with one or more predetermined adaptive types, and the step of determining the adaptive font based on the one or more determined feature parameters of the one or more important voice features includes the step of applying the one or more predetermined adaptive types based on the one or more determined feature parameters of the one or more important voice features.

8. The method according to claim 7, further comprising the step of determining an application coefficient of a predetermined adaptive type to each of the one or more important speech features based on the determined feature parameters associated with the important speech feature, wherein the step of applying the one or more predetermined adaptive types based on the one or more determined feature parameters of the one or more important speech features includes the step of applying the one or more predetermined adaptive types to a predefined font in a proportion that correlates with the application coefficient of the one or more predetermined adaptive types in order to generate the adaptive font.

9. The method according to claim 8, wherein for each of the one or more important voice features, the application coefficient of the predetermined adaptive type for each of the aforementioned important voice features lies on a scale having a minimum and a maximum value, the minimum value indicating a first degree of applicability of the predetermined adaptive type, and the maximum value indicating a second degree of applicability of the predetermined adaptive type.

10. The method according to claim 8 or 9, wherein the predefined font is a variable font to which one or more embodiments can be progressively adapted.

11. The method according to any one of claims 1 to 5, wherein the step of determining the adaptive font based on the plurality of feature parameters includes the step of selecting a substitute font as the adaptive font from a plurality of substitute fonts based on the plurality of feature parameters.

12. The method according to claim 11, wherein the plurality of proxy fonts correspond to a plurality of predetermined fonts, and each of the plurality of proxy fonts is associated with a predetermined set of values ​​of the feature parameters for the plurality of speech features.

13. The method of claim 12, wherein the step of selecting a proxy font from a plurality of proxy fonts as the adaptive font based on the plurality of feature parameters includes the step of selecting a proxy font from the plurality of proxy fonts whose predetermined set of values ​​is closest to the plurality of feature parameters.

14. The method according to claim 11, wherein the plurality of proxy fonts correspond to a plurality of predetermined fonts generated for at least a range of discrete values ​​for each of the one or more feature parameters of the one or more important speech features, each proxy font is associated with a specific combination of discrete values ​​that form its respective proxy coordinate set, and the step of selecting the proxy font from the plurality of proxy fonts as the adaptive font based on the plurality of feature parameters includes the step of selecting the proxy font from the plurality of proxy fonts whose proxy coordinate set is closest to the one or more determined feature parameters of the one or more important speech features.

15. The method according to any one of claims 1 to 5, wherein the step of determining the adaptive font based on the plurality of feature parameters further includes the step of using a trained model to generate the adaptive font based on the plurality of feature parameters.

16. The method according to claim 15, wherein the further-trained model is a model trained on a set of labeled fonts each representing several specific speech features in order to determine reference associations between a set of combinations of values ​​of the plurality of feature parameters and each set of derived reference fonts, and the step of using the further-trained model to generate the adaptive font based on the plurality of feature parameters includes the step of generating the adaptive font based on the plurality of feature parameters and the reference associations.

17. The method according to claim 16, wherein the step of generating the adaptive font based on the plurality of feature parameters and the reference associations includes the step of applying to one or more fonts from the derived set of reference fonts one or more adaptations of one or more microtypographical modes or one or more microtypographical elements, one or more adaptations of one or more macrotypographical modes or one or more macrotypographical elements, one or more adaptations of one or more display characteristics or one or more display elements, and one or more adaptations of one or more dynamic characteristics.

18. The method according to claim 16 or 17, wherein the step of generating the adaptive font based on the plurality of feature parameters and the reference associations includes the step of generating the adaptive font using either a stable diffusion process or a character skeleton mapping process based on the plurality of feature parameters and the reference associations.

19. The method according to claim 6, wherein the step of determining the adaptive font based on the one or more determined feature parameters of the one or more important sound features includes the step of applying one or more custom adaptive types to the custom font based on the one or more determined feature parameters of the one or more important sound features.

20. The method according to claim 19, wherein the one or more custom adaptation types are defined according to one or more custom rules for the custom font.

21. The method according to any one of the preceding claims, wherein the utterance is a unit of utterance.

22. The method according to claim 21, wherein the plurality of feature parameters associated with the plurality of speech features determined for the utterance correspond to a first plurality of feature parameters, the method further comprises: determining one or more subunits of utterance in a unit of utterance having a second plurality of feature parameters each which substantially deviates from the first plurality of feature parameters with respect to at least one speech feature; and determining a modified adaptive font for the one or more subunits of utterance based on a quantified deviation between the second plurality of feature parameters and the first plurality of feature parameters, and the method further comprises formatting the at least portion of the text information based on the adaptive font for display to a user, the method comprising formatting one or more portions of the text information corresponding to the one or more subunits of utterance using the modified adaptive font.

23. The method according to any one of the preceding claims, wherein the method is for conveying the oral characteristics of the speech of the audio data to a person with a hearing impairment or hearing loss.

24. The method according to any one of the preceding claims, further comprising the step of determining the audio data from one or more of the following: video, audio files, audio broadcasts, phone calls, live audio events or virtual audio events, or video conferences.

25. One or more computing devices comprising one or more processors configured to perform the method described in any one of the preceding claims.

26. One or more computer-readable storage media that, when executed by one or more processors, stores instructions causing the one or more processors to perform the method according to any one of claims 1 to 24.

Images