System for deep learning-based sentiment analysis in spoken texts

Abstract

A system for sentiment analysis in language texts based on deep learning, consisting of: a preprocessing unit configured to receive and normalize raw text input from various sources, with the unit performing tokenization, punctuation removal, text stemming, and standardization to prepare text data for analysis; an embedding unit that is operationally connected to the preprocessing unit and is configured to convert normalized text data into numerical representations, wherein the embedding unit also contains pre-trained language embeddings to capture semantic relationships in the text and dynamically selects embeddings based on the source and linguistic context of the text data; a sentiment analysis neural network (SANNet) operationally connected to the embedding unit, comprising layers of a convolutional neural network (CNN) for local pattern recognition and layers of a recurrent neural network (RNN) configured to capture sequential dependencies within text data, with the SANNet also including an attention mechanism to dynamically weight word relevance based on contextual meaning, and the SANNet being trained to interpret and classify complex sentiment patterns in the text; A context module integrated into the neural network for mood analysis. The module includes a bidirectional transformer layer for the simultaneous processing of forward and backward word dependencies, enabling the system to capture complex language structures, including sarcasm, irony, and mixed emotions, and thus perform an accurate mood classification; an output interpretation unit that is operationally connected to the neural network for sentiment analysis and is configured to convert probability-based classification results into human-readable sentiment categories with associated confidence levels, wherein the output interpretation unit also provides interface compatibility for external applications, thus enabling direct integration of sentiment classifications; A continuous training module connected to the neural network for sentiment analysis. The module is configured to use real-time user feedback and sentiment accuracy metrics to retrain the SANNet, thus ensuring adaptive learning for evolving language patterns and domain-specific contexts; a GPU / TPU-optimized processing card that is operationally connected to the neural network for sentiment analysis and is configured to accelerate deep learning calculations within the system, thereby enabling high-throughput, real-time sentiment analysis.

Description

Field of the invention

[0001] Specifically, it aims to: Improve the accuracy of classifying human sentiment in text data by leveraging a collection of domain-specific deep neural networks trained on universal corpora in multiple languages ​​and optimized for different types of text sources (social media content, news articles, customer feedback, etc.). Improve pre-trained sentence embedding models specifically optimized for social media sentiment analysis. Background of the invention

[0002] Sentiment analysis using natural language processing (NLP) has emerged as the most popular application for analyzing public opinion, customer satisfaction, and social trends. Conventional sentiment analysis methods are often too error-prone to be used effectively because human language is simply too complex and diverse. Often, this use of shallow learning models or more traditional approaches such as bag-of-words and lexicons cannot capture the nuances influenced by chance or idioms as an end-to-end solution. The most significant contribution is a deep learning sentiment analysis system that uses stacked neural architectures to predict emotional response to text. It was designed for both high-resolution surrealism and a better understanding of complex words in diverse text sources.

[0003] Texts containing human emotions are not well understood by machines, but advances in recent years have made the field of sentiment analysis a central component of natural language processing (NLP). This capability can be highly beneficial in customer service, social media monitoring, and trend analysis, as knowledge of user sentiment can provide useful insights for both businesses and politicians. The lexicon-based method, which relies on predefined lists of positive and negative words to classify sentiment, is an old traditional approach to sentiment analysis. In lexicon-based systems, sentiment is calculated based on the presence of certain words or phrases in a text and their values ​​are summed to determine how positive / negative they were. While this idea is nice and simple, it doesn't account for the complexity of language in the real world.Because words themselves can have more than one meaning depending on the context, lexicon-based methods often misclassify non-obvious moods (such as sarcasm), negations, and other text parodies too frequently to ensure the high degree of generalization required in the final classification phase, resulting in lower accuracy rates. Furthermore, these methods are highly lexicon-oriented and therefore less flexible with regard to changing language trends or the use of domain-specific terminology.

[0004] This necessitated the introduction of machine learning techniques to mitigate various limitations of dictionary-based sentiment analysis. The use of algorithms such as Naive Bayes, Support Vector Machines (SVM), and decision trees are just three such methods for classifying texts based on sentiment. Sentiment Analysis Using Machine Learning: In machine learning-based sentiment analysis, the system is trained using patterns available in labeled data, where each text entry is assigned a sentiment label, i.e., positive, negative, or neutral. Using features extracted from the text (word frequency, part-of-speech tags, and n-grams), machine learning models can learn patterns of expressions related to specific sentiments and then apply these rules to new, previously unseen text.We found that the former model significantly outperforms many dictionary-based methods, but also has some limitations. Traditional machine learning models, on the other hand, require extensive feature engineering, meaning domain experts carefully select features or staff spend hours fine-tuning to capture sentiment-related patterns in text. This is a very tedious process and often leads to failure due to the parallelism of complex meanings within sentences that are overlaid with implicit meanings. Finally, it is difficult to generalize across different domains or even languages ​​with such models. For example, sentiment patterns in social media posts are significantly different from those in formal news articles. A model trained in one domain may not perform as well in another.Due to the nature of machine learning, models must revisit training data and frequently need to be retrained and optimized in a domain-specific manner, which can sometimes be resource-intensive.

[0005] Sentiment analysis made further advances with the introduction of deep learning, which enabled more complex neural network architectures to capture semantic content in high-dimensional, abstract texts. Sentiment analysis was one of the first areas to apply deep learning models, as CNNs and LSTMs have their own advantages. Convolutional neural networks (CNNs) were originally developed for image representation and have been successfully used to capture local patterns in text sequences, such as frequently co-occurring words or phrases that express sentiment. However, CNNs are unable to handle the sequential dependencies between words in sentences, and significant semantic relationships such as sentiment between them can involve more complicated long-distance relationships involving many other words in a sentence, which is why they often struggle.A recurrent neural network (RNN) and its variants such as long short-term memory networks, known as LSTMs or gated recurrent units (GRUs), are specifically designed for this reason: to process sentences in sequential order, retaining some previously presented information while looking at other words. This ability to retain context makes RNNs a suitable choice when it comes to sentiment analysis with difficult sentence structure. For this reason, they have some disadvantages, the most important of which is computational power (it is difficult for RNNs with many steps). Training RNN models is computationally intensive, and their performance can suffer when long text sequences cause the vanishing gradient problem, which makes it difficult for the model to propagate information across many time steps.

[0006] Transformer-based models have significantly improved sentiment analysis in recent years by adding attention mechanisms to focus on different parts of a piece of text, such as with BERT (Bidirectional Encoder Representations from Transformers) or GPT (Generative Pre-trained Transformer). Transformers address the limitations of RNNs' sequential processing, enabling parallelization and giving them the ability to capture both near and distant dependencies within a sentence. This is further supported by models such as BERT, which has also demonstrated state-of-the-art results in many NLP tasks, including sentiment analysis, as its bidirectional nature allows a model to have context in both the forward and backward directions

[15] .This type of directionality aids in understanding complex linguistic structures and improves meaning, allowing it to depend not only on previous words but also on future n-1 tokens (where n is the sequence length). Although transformer models are more accurate, they still have limitations in many ways. One of the challenges with their use is that they are computationally intensive, especially in real-time applications where latency and resource constraints prevent them from operating on heavily loaded networks. Using transformer-based models in production requires large amounts of computational resources such as GPUs, TPUs, etc., which may be inaccessible to smaller organizations or applications that require lightweight solutions.Furthermore, transformer models can easily be overfit, especially when trained with small and unbalanced datasets, which would distort the new sentiment.

[0007] The first common limitation that affects all current solutions is the inability to understand subtle emotions. First and foremost, an emotion is never truly binary when expressed in any language; ordinary people simply don't understand all the countless words in between. Furthermore, an emotion can involve more complex emotions or meanings, such as sarcasm, irony, or simultaneous positive and negative feelings. These emotional manifestations are difficult for most of these methods to achieve in 2021, which is why they make errors that can influence business decisions. Next, multilingual sentiment analysis becomes more influential, but most models are based on English texts because the training data is linguistically accessible. The problem with this approach is that a model trained on English data simply won't work on non-English data.Some models are language-specific but require different training and maintenance, which could also complicate implementation for companies. Domain adaptability is another broad issue. Not all sentiment is the same, and even the best; most generalized models show limited accuracy in domains they've never "seen" before. The problem is further exacerbated by transfer learning, the allelopathy of knowledge transfer between two similar, but not too similar, instances. Pre-trained language models are not future-proof, meaning they can't understand colloquialisms, memes, or archaisms used on Twitter.

[0008] In summary, there have been tremendous improvements in sentiment analysis thanks to new NLP developments and deep learning, but the prevailing solutions still face many problems. Traditional dictionary-based methods and ML techniques fail to perform accurate sentiment analysis because they rely on predefined rules or shallow learning algorithms. While their deep learning counterparts are more powerful, they often require many resources and do not cope well with subtle (or domain-specific) sentiments. Transformer-based models have shown the most promise, but are computationally intensive and may not generalize well to other languages ​​or domains.Human language is a complex system combining semantics and contextual understanding; this, along with the requirement for scalable, real-time sentiment analysis across diverse contexts, presents severe limitations that require a comprehensive solution to address these drawbacks. Summary of the invention

[0009] The invention relates to a system comprising i) an application with a deep learning-based solution that uses specially developed neural network models to analyze sentiment in text data. This architecture comprises various components, such as a preprocessing block, an embedding layer, and the Sentiment Analysis Neural Network (SANNet); it also includes a context module for context-dependent feature enrichment represented by word vectors in both languages, and finally, an output interpretation. These modules are tightly coupled and efficiently perform sentiment classification by transforming raw text data into a structured format, enabling very precise classification as positive, negative, or neutral.The system is further fine-tuned with a training module that leverages a large multilingual sentiment dataset and a feedback mechanism that makes the model increasingly more accurate in real time, depending on how well it can correctly assign sentiments. The hardware stack consists of a processing unit, a GPU / TPU-optimized board for deep learning calculations, and memory that stores model training data and sentiment-related datasets.

[0010] One or more advantages of the present invention are to provide a powerful deep learning-based system applicable to sentiment analysis of speech texts with high stability and versatility, avoiding the aforementioned problems. Powerful neural network architectures enable the system to accurately identify nuanced sentiments in a variety of text sources (social media posts, product reviews and customer feedback, or news articles). Using this method, it can capture subtle contextual cues and nuances that many conventional lexicon-based machine learning models miss.The present invention also aims to develop a new sentiment analysis framework that implements transformer-based deep learning models with attention mechanisms, enabling this system's language model to process text data both non-sequentially and bidirectionally. This proved helpful in capturing nuances and dependencies across large text sequences, ultimately leading to a better overall sentiment classification score.

[0011] Furthermore, the invention aims to provide a sentiment analysis system capable of processing large amounts of text at full speed and can therefore be used where immediate insights into public opinion or customer satisfaction trends are required. A powerful GPU- or TPU-optimized processing module could be embedded to enable high-throughput data processing without sacrificing speed or accuracy, thereby providing timely updates to sentiment results. Another objective of the invention is to provide a method for performing multilingual sentiment analysis within a single framework, helping to understand sentiments between different languages ​​and their dialects.The system is quite versatile as different languages ​​can be supported through appropriate embedding and fine-tuning mechanisms, making it a solution for international sentiment analysis problems.

[0012] The present invention further addresses domain-specific needs of sentiment analysis by providing transfer learning and fine-tuning layers that allow the model to adapt to different contexts with almost no additional training effort. This flexibility means the system can be trained to provide highly relevant classifications of sentiment across different sectors or for specific text types, making it suitable for a wide range of industry applications. This optimization of the deep learning model's architecture and the use of an efficient storage solution enable scaling for large and small organizations while minimizing resource requirements.By achieving these goals, the solution becomes a complete sentiment analysis system with trade-offs between accuracy and addictive performance; overall, it is an ideal tool for understanding all types of sentiment from complex language texts across applications and domains. SHORT DESCRIPTION OF THE FIGURE

[0013] These and other features, aspects, and advantages of the present invention will become more readily understood when the following detailed description is read in conjunction with the accompanying drawings, in which like characters represent like parts throughout. Fig. Figure 1 shows a block diagram of a system for deep learning-based sentiment analysis in speech texts.

[0014] In Fig.Figure 1 shows a block diagram of a system for deep learning-based sentiment analysis in speech texts. The system 100 includes: a preprocessing unit 102 configured to receive and normalize raw text inputs from various sources, wherein the unit performs tokenization, punctuation removal, stemming, and standardization to prepare text data for analysis; an embedding unit 104 operatively connected to the preprocessing unit and configured to convert normalized text data into numerical representations, wherein the embedding unit further includes pre-trained language embeddings to capture semantic relationships in the text and dynamically selects embeddings based on the source and linguistic context of the text data;a sentiment analysis neural network (SANNet) 106 operatively connected to the embedding unit and comprising convolutional neural network (CNN) layers for local pattern recognition and recurrent neural network (RNN) layers 106(a) configured to capture sequential dependencies within text data, the SANNet further comprising an attention mechanism 106(b) to dynamically weight word relevance based on contextual meaning, the SANNet being trained to interpret and classify complex sentiment patterns in the text;a contextual module 108 integrated with the neural network for sentiment analysis, the module comprising a bidirectional transformer layer for simultaneously processing forward and backward word dependencies, enabling the system to capture complex language structures, including sarcasm, irony, and mixed emotions, to achieve accurate sentiment classification; an output interpretation unit 110 operatively connected to the neural network for sentiment analysis and configured to convert probability-based classification results into human-readable sentiment categories with associated confidence scores, the output interpretation unit also providing interface compatibility with external applications, thus enabling direct integration of sentiment classifications;a continuous training module 112 connected to the sentiment analysis neural network, the module configured to use real-time user feedback and sentiment accuracy metrics to retrain the SANNet, ensuring adaptive learning for evolving speech patterns and domain-specific contexts; and a GPU / TPU-optimized processing card 114 operatively connected to the sentiment analysis neural network and configured to accelerate deep learning computations within the system, thereby enabling real-time, high-throughput sentiment analysis.

[0015] In one embodiment, the preprocessing unit 102 is further configured to analyze text length and syntactic structure to apply differential normalization processes, selectively prioritizing slang, idioms, or domain-specific jargon relevant to the text source, thereby ensuring optimized input preparation for domain-specific sentiment contexts.

[0016] In one embodiment, the embedding unit 104 is further configured to use a two-layer embedding selection process that evaluates the contextual frequency and semantic distance of words within the input text and dynamically integrates embeddings from various models such as Word2Vec, GloVe, and BERT based on the linguistic features and sentiment patterns detected in the input.

[0017] In one embodiment, the 106-CNN layer of the sentiment analysis neural network is designed to capture hierarchical n-grams by implementing multiple convolutional filters with different receptive fields, enabling fine-grained analysis of syntactic and lexical features associated with different sentiment intensities in the text data.

[0018] In one embodiment, the RNN layer of neural network 106(a) uses Gated Recurrent Units (GRUs) for sentiment analysis, which are specifically optimized for managing long-range dependencies in sentences, where the GRUs also incorporate temporal dropout techniques to reduce overfitting and maintain accuracy for variable-length text inputs.

[0019] In one embodiment, the attention mechanism 106(b) within the SANNet is further configured to apply adaptive weights based on domain and context sensitivity, leveraging a cross-layer of attention to improve the accuracy of sentiment recognition in scenarios with mixed or ambiguous emotions.

[0020] In one embodiment, the bidirectional transformer layer 108 of the context module also integrates a language domain detection sub-module that identifies language-specific constructs or idiomatic expressions and assigns them certain weights, thereby improving sentiment classification in multilingual datasets.

[0021] In one embodiment, the output interpretation unit 110 is configured to analyze the confidence scores of sentiment classifications and apply a hierarchical sentiment interpretation scheme, dynamically adjusting the granularity of the sentiment label based on the confidence threshold, allowing the system to offer refined sentiment categories for high-confidence results and generalized sentiment categories for lower-confidence classifications.

[0022] In one embodiment, the continuous training module 112 further includes a drift detection mechanism that evaluates changes in linguistic patterns over time, dynamically identifies shifts in sentiment expressions, and adjusts the composition of the training dataset to address new language trends without manual intervention.

[0023] In one embodiment, the GPU / TPU-optimized processing card 114 includes a load balancing subsystem that dynamically adjusts resource allocation based on processing demand and prioritizes compute-intensive tasks related to real-time sentiment analysis during peak data inflow times, thereby maintaining system performance stability.

[0024] A deep learning system for sentiment analysis is divided into several linked components, each designed to perform specific tasks necessary to determine the true nature of the emotions behind text data. The processing core, including the preprocessing unit, accepts raw text input from multiple sources such as social media feeds, chat interfaces, and document repositories. This module performs essential functions such as tokenization, filtering out unwanted punctuation, stemming, and text normalization. These are preprocessing jobs that add structure to the raw input data so that it can later be analyzed by downstream components that perform sentiment analysis in a more unified manner.

[0025] After the preprocessing steps, the next step is to pass this data to the embedding layer, which transforms our structured text data into numerical form using embeddings from previously trained models such as Word2Vec, Glove, or BERT (SOTA embeddings). Such embeddings enable the model to understand the context, nuances, and meaning inherent in natural language words / phrases, coupled with high-precision sentiment analysis. The embedding unit is designed to dynamically select embeddings based on the context and origin of the text, personalizing the input interpretation through targeted model understanding.

[0026] This sends the resulting text to a deep neural network classifier for sentiment analysis, which internally uses a convolution neural network (CNN) and at least one long short-term memory or gated recurrent unit (LSTM / GRU). The CNN layers in SANNet capture local syntactic patterns within sentences, and the RNN layer interprets sequential dependencies, allowing this system to understand complex sentence structures and dependencies. This process is further refined by a context module that includes attention mechanisms for the model to identify specific parts of the text that reveal more about what sentiment it is intended to convey. This attention mechanism is particularly good at capturing subtle sentiment cues such as sarcasm, irony, and mixed emotions (all very difficult to model with conventional models).

[0027] Finally, the output interpretation unit uses a neural network such as HTL to interpret probabilistic NLP results and generate human-readable sentiment classification results. This layer is configured to return confidence scores for each classification, giving us insight into how the model is performing. The results can then be delivered to user interfaces or connected directly to applications and stored in a system storage unit for later use.

[0028] The system also consists of a persistent training module that enables real-time refinement through feedback in SANNet. This generates user-friendly yet complex, model-based feedback using models that are updated (approximately) daily, depending on perceived variations in language and cultural sentiment signals extracted through user interaction with the system. The training module is housed in a GPU / TPU-optimized processing board, accelerating the generation of new models to enable timely model updates.

[0029] A device architecture with a CPU to manage data flow between components, a GPU or TPU for hardware acceleration of deep learning operations, and memory that stores both static and dynamic data; it stores trained model parameters, embeddings, and historical sentiment values. The processing board features a GPU / TPU-based compute engine connected to the CPU via a high-speed data bus, enabling efficient data transfer between the various subcomponents on the board, resulting in lower overall latency.

[0030] The present invention relates to the technical field of natural language processing (NLP) and artificial intelligence, particularly with respect to an application using sentiment analysis for text data using deep learning techniques. It consists of the creation or fine-tuning of transformer models, attention mechanisms, and real-time data processing frameworks for sentiment analysis of multilingual texts. This invention applies to all use cases requiring sophisticated user sentiment analysis (e.g., social media monitoring, CRM, trend detection, and content moderation), where accurate real-time detection along with fine-grained categorization of user sentiment is crucial. REFERENCES 100 A system for deep learning-based sentiment analysis in speech texts. 102 A preprocessing unit 104 Embedding unit 106 Neural Network for Sentiment Analysis (Sannet) 106a Recurrent Neural Networks (RNN) Layers 106b Attention mechanism 108 A context module 110 Output interpretation unit 112 Further training module 114 \ GPU / TPU-optimized processing card

Claims

[1] A system for deep learning-based sentiment analysis in speech texts, consisting of: a preprocessing unit configured to receive and normalize raw text inputs from various sources, wherein the unit performs tokenization, punctuation removal, stemming, and standardization to prepare text data for analysis; an embedding unit operatively connected to the preprocessing unit and configured to convert normalized text data into numerical representations, wherein the embedding unit further includes pre-trained language embeddings to capture semantic relationships in the text and dynamically selects embeddings based on the source and linguistic context of the text data; a sentiment analysis neural network (SANNet) operatively connected to the embedding unit and comprising convolutional neural network (CNN) layers for local pattern recognition and recurrent neural network (RNN) layers configured to capture sequential dependencies within text data, wherein the SANNet further includes an attention mechanism to dynamically weight word relevance based on contextual meaning, wherein the SANNet is trained to interpret and classify complex sentiment patterns in the text; A context module integrated into the neural network for sentiment analysis. This module includes a bidirectional transformer layer for simultaneous processing of forward and backward word dependencies, allowing the system to capture complex language structures, including sarcasm, irony, and mixed emotions, and thus perform accurate sentiment classification. an output interpretation unit operatively connected to the neural network for sentiment analysis and configured to convert probability-based classification results into human-readable sentiment categories with associated confidence scores, wherein the output interpretation unit also provides interface compatibility with external applications, thus enabling direct integration of sentiment classifications; A continuous training module connected to the neural network for sentiment analysis. The module is configured to leverage real-time user feedback and sentiment accuracy metrics to retrain the SANNet, ensuring adaptive learning for evolving language patterns and domain-specific contexts. a GPU / TPU-optimized processing card operatively connected to the sentiment analysis neural network and configured to accelerate the deep learning computations within the system, thereby enabling real-time, high-throughput sentiment analysis. [2] The system of claim 1, wherein the preprocessing unit is further configured to analyze text length and syntactic structure to apply differential normalization processes, selectively prioritizing slang, idioms, or subject-specific jargon relevant to the text source, thereby ensuring optimized input preparation for subject-specific sentiment contexts. [3] The system of claim 1, wherein the embedding unit is further configured to use a two-layer embedding selection process that evaluates the contextual frequency and semantic distance of words within the input text and dynamically integrates embeddings from different models such as Word2Vec, GloVe and BERT based on the linguistic features and sentiment patterns detected in the input. [4] The system of claim 1, wherein the CNN layer of the neural network for sentiment analysis is designed to capture hierarchical N-grams by implementing multiple convolutional filters with different receptive fields, thereby enabling fine-grained analysis of syntactic and lexical features associated with different sentiment intensities in the text data. [5] The system of claim 1, wherein the RNN layer of the neural network for sentiment analysis uses Gated Recurrent Units (GRUs) specifically optimized for managing long-range dependencies in sentences, the GRUs further incorporating temporal dropout techniques to avoid overfitting and maintain accuracy for variable-length text inputs. [6] The system of claim 1, wherein the attention mechanism within the SANNet is additionally configured to apply adaptive weights based on domain and context sensitivity, leveraging a cross-layer of attention to improve the accuracy of sentiment recognition in scenarios with mixed or ambiguous emotions. [7] The system of claim 1, wherein the bidirectional transformer layer of the context module further integrates a language domain recognition sub-module that identifies language-specific constructs or idiomatic expressions and assigns them certain weights, thereby improving sentiment classification in multilingual datasets. [8] The system of claim 1, wherein the output interpretation unit is configured to analyze the confidence scores of sentiment classifications and apply a hierarchical sentiment interpretation scheme, dynamically adjusting the granularity of the sentiment labels based on the confidence threshold, whereby the system can offer refined sentiment categories for high-confidence results and generalized sentiment categories for lower-confidence classifications. [9] The system of claim 1, wherein the continuous training module further comprises a drift detection mechanism that evaluates changes in linguistic patterns over time, dynamically identifies shifts in sentiment expressions, and adjusts the composition of the training dataset to address new language trends without manual intervention. [10] The system of claim 1, wherein the GPU / TPU-optimized processing card includes a load balancing subsystem that dynamically adjusts resource allocation based on processing demand and prioritizes compute-intensive tasks related to real-time sentiment analysis during peak data inflow times, thereby maintaining system performance stability.

Images