A text data preprocessing method suitable for a financial large model

Abstract

The application relates to the technical field of artificial intelligence, and discloses a text data preprocessing method suitable for a financial large model, which comprises the following steps: receiving an original financial text sequence sorted by time stamp; performing word segmentation and part-of-speech tagging; dynamically calculating an emotional fluctuation index of each text unit based on a sliding time window; adaptively setting an emotional noise judgment threshold and marking a candidate noise unit; evaluating semantic stability through embedding space cosine similarity, and secondarily confirming noise; after the confirmed noise is removed, performing reference resolution and logical connection reconstruction to repair context coherence; and outputting a pure text sequence. The system comprises corresponding function modules. Through the dynamic threshold and semantic double verification mechanism, the application significantly improves the effective information retention rate and noise removal accuracy in a high-noise scene, and guarantees the quality and processing throughput of input data of the financial large model.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence, specifically relating to a text data preprocessing method suitable for large financial models. Background Technology

[0002] With the increasing application of large-scale financial models in scenarios such as intelligent investment research, risk warning, and public opinion analysis, the real-time preprocessing capability of high-quality text data has become a key support for model performance. Text data in the financial field is characterized by high timeliness, strong semantic sensitivity, and complex noise structures. Especially during periods of sharp market fluctuations, news feeds, social media, and financial forums often see a surge of sudden, emotionally charged content, including unverified rumors, repeatedly circulated panic information, or maliciously manipulated statements. While such noise lacks real fundamental value, its high-frequency diffusion can easily interfere with the model's accurate judgment of market sentiment, thereby affecting the reliability of downstream decision-making tasks.

[0003] Text data preprocessing methods suitable for large financial models focus on efficiently extracting effective signals from massive heterogeneous information sources and suppressing the interference of irrational noise. The core objective is to dynamically identify and filter emotional disturbances unrelated to the actual market state while ensuring low-latency processing, retaining original text units with added information, thereby providing large models with clean, stable, and semantically consistent input sequences.

[0004] Noise filtering employs rules based on fixed thresholds or static machine learning models, such as setting upper limits for keyword frequency, absolute thresholds for sentiment polarity, or deduplication coefficients for similarity. However, such methods exhibit severe limitations when faced with sudden changes in noise intensity caused by unexpected market events: on the one hand, fixed thresholds cannot perceive real-time changes in the external financial environment, leading to excessive deletion of valid text containing genuine warning signals during periods of high volatility; on the other hand, they may miss low-frequency but highly harmful hidden rumors during periods of low volatility and stability. More importantly, existing solutions do not couple the noise identification mechanism with objective indicators of the financial market (such as the VIX fear index, stock price volatility, or sudden increases in trading volume), causing the preprocessing process to be detached from the actual market context and making it difficult to achieve adaptive adjustment of noise discrimination standards. Summary of the Invention

[0005] This invention provides a text data preprocessing method suitable for large-scale financial models, aiming to solve the technical problems of sudden market sentiment noise in real-time financial news streams causing a sharp drop in preprocessing efficiency, and existing methods relying on fixed threshold filtering failing to dynamically adapt to sudden changes in noise intensity, resulting in the accidental deletion of valid information. The method constructs a multi-level dynamic noise perception and adaptive filtering mechanism, combined with semantic stability assessment and contextual consistency verification, to achieve accurate identification and efficient removal of sudden sentiment noise in financial text streams, while retaining valid text fragments with real market signal value.

[0006] This invention provides a text data preprocessing method suitable for large-scale financial models, comprising:

[0007] Receive raw text sequences from a real-time financial news stream, the raw text sequences consisting of multiple text units sorted by timestamp;

[0008] The original text sequence is initially segmented and part-of-speech tagged to generate a word sequence with grammatical attributes;

[0009] Based on the word sequence, the sentiment fluctuation index of each text unit is calculated. The sentiment fluctuation index is determined by the deviation between the rate of change of sentiment word frequency density within a sliding time window and the historical baseline sentiment level.

[0010] Based on the emotional fluctuation index, the emotional noise judgment threshold for the current time window is dynamically set, and the emotional noise judgment threshold is adaptively adjusted according to the standard deviation of the emotional fluctuation index within the sliding time window.

[0011] Text units whose emotional fluctuation index exceeds the emotional noise determination threshold are marked as candidate noise units;

[0012] The semantic stability of the candidate noise unit is evaluated by calculating the mean cosine similarity between the unit and its adjacent non-noise units in the embedding space.

[0013] If the mean cosine similarity is lower than the preset semantic consistency lower limit, the candidate noise unit is identified as emotional noise and is removed.

[0014] The text sequence after removing emotional noise is subjected to contextual coherence restoration, which includes the resolution of references and the reconstruction of logical connections at the text boundaries before and after the removed unit.

[0015] Output the repaired clean text sequence for use in subsequent training or inference of large financial models.

[0016] As one embodiment of the present invention, the calculation of the sentiment fluctuation index of each text unit specifically includes:

[0017] The system matches emotional keywords in the current text unit from a pre-defined financial sentiment dictionary, which includes a positive sentiment word set and a negative sentiment word set, with each type of word labeled with an emotional intensity weight.

[0018] The raw emotion score is obtained by summing the emotion intensity weights of all emotion keywords in the current text unit.

[0019] Using a sliding time window of 30 text units, the moving average of the original sentiment score within the current window is calculated as the historical baseline sentiment level.

[0020] Calculate the absolute difference between the original sentiment score of the current text unit and the historical baseline sentiment level, and divide it by the historical baseline sentiment level to obtain the relative sentiment deviation.

[0021] The relative emotional deviation is then subjected to exponential smoothing to obtain the final emotional fluctuation index.

[0022] As one embodiment of the present invention, the dynamic setting of the emotional noise determination threshold for the current time window specifically includes:

[0023] Within the sliding time window, calculate the sample standard deviation of the sentiment fluctuation index for all text units;

[0024] Multiply the sample standard deviation by a preset sensitivity coefficient, wherein the sensitivity coefficient is 1.5;

[0025] The dynamic emotion noise threshold is obtained by adding the historical baseline emotion level to the product.

[0026] As one embodiment of the present invention, the semantic stability evaluation of candidate noise units specifically includes:

[0027] Using a pre-trained word embedding model in the financial domain, candidate noisy units and their five forward and five backward non-noisy text units are mapped to a unified vector space.

[0028] Average pooling is performed on the vector representation of each non-noise text unit to obtain its context semantic vector;

[0029] Calculate the cosine similarity between the semantic vector of the candidate noise unit and the semantic vector of each context;

[0030] The arithmetic mean of all cosine similarities is taken as the semantic stability index of the candidate noise unit.

[0031] As one embodiment of the present invention, the preset semantic consistency lower limit is 0.35.

[0032] As one embodiment of the present invention, the context coherence repair specifically includes:

[0033] Identify pronouns or omitted subjects at the end of the text unit preceding the noise-removed unit;

[0034] Extract latent referential entities from forward-looking, non-noise text units;

[0035] Based on entity co-occurrence frequency and syntactic dependency relationship, the most likely referent is selected for explicit replacement;

[0036] Logical relationship analysis is performed on the conjunctions in the sentences before and after the removed position. If there is a break in the causal, adversative, or progressive relationship, standardized cohesive phrases are inserted to restore the coherence of the discourse.

[0037] As one embodiment of the present invention, the word embedding model in the financial field is a continuous bag-of-words model trained on a corpus of 1 billion financial news articles, with a vector dimension of 300, a training window size of 8, and a negative sampling number of 15.

[0038] As one embodiment of the present invention, the financial sentiment dictionary is constructed in the following manner:

[0039] Collect news texts published by authoritative financial media in the past 5 years with accompanying market reaction tags;

[0040] Inferring the polarity of textual sentiment by using the magnitude of event-driven stock price fluctuations;

[0041] Frequently occurring emotional expressions are manually verified and their intensity is calibrated.

[0042] A structured dictionary containing 3,200 emotion terms was created, including 1,400 positive terms and 1,800 negative terms. The emotion intensity weight of each term ranges from 0.1 to 1.0.

[0043] This invention provides a text data preprocessing system suitable for large-scale financial models, comprising:

[0044] The raw text receiving module is used to receive raw text sequences from real-time financial news streams;

[0045] The syntax parsing module is used to perform initial word segmentation and part-of-speech tagging on the original text sequence to generate a word sequence with grammatical attributes;

[0046] The emotion fluctuation calculation module is used to calculate the emotion fluctuation index of each text unit based on the word sequence;

[0047] The dynamic threshold setting module is used to dynamically set the emotional noise judgment threshold for the current time window based on the emotional fluctuation index.

[0048] A noise candidate labeling module is used to label text units whose emotional fluctuation index exceeds the emotional noise determination threshold as candidate noise units;

[0049] A semantic stability evaluation module is used to evaluate the semantic stability of the candidate noise units;

[0050] The noise identification and elimination module is used to identify candidate noise units as emotional noise and eliminate them when the semantic stability evaluation result is lower than the preset semantic consistency lower limit.

[0051] The context restoration module is used to restore the contextual coherence of text sequences after removing emotional noise.

[0052] The Clean Text Output module is used to output the repaired clean text sequence.

[0053] In one embodiment of the present invention, the emotion fluctuation calculation module is specifically configured as follows:

[0054] Match sentiment keywords in the current text unit from a pre-defined financial sentiment dictionary;

[0055] The raw emotion score is obtained by summing the emotion intensity weights of all emotion keywords in the current text unit.

[0056] Using a sliding time window of 30 text units, the moving average of the original sentiment score within the current window is calculated as the historical baseline sentiment level.

[0057] Calculate the absolute difference between the original sentiment score of the current text unit and the historical baseline sentiment level, and divide it by the historical baseline sentiment level to obtain the relative sentiment deviation.

[0058] The relative emotional deviation is then subjected to exponential smoothing to obtain the final emotional fluctuation index.

[0059] In one embodiment of the present invention, the dynamic threshold setting module is specifically configured as follows:

[0060] Within the sliding time window, calculate the sample standard deviation of the sentiment fluctuation index for all text units;

[0061] Multiply the sample standard deviation by a sensitivity coefficient of 1.5;

[0062] The dynamic emotion noise threshold is obtained by adding the historical baseline emotion level to the product.

[0063] As one embodiment of the present invention, the semantic stability evaluation module is specifically configured as follows:

[0064] Using a pre-trained word embedding model in the financial domain, candidate noisy units and their five forward and five backward non-noisy text units are mapped to a unified vector space.

[0065] Average pooling is performed on the vector representation of each non-noise text unit to obtain its context semantic vector;

[0066] Calculate the cosine similarity between the semantic vector of the candidate noise unit and the semantic vector of each context;

[0067] The arithmetic mean of all cosine similarities is taken as the semantic stability index of the candidate noise unit.

[0068] In one embodiment of the present invention, the context repair module is specifically configured as follows:

[0069] Identify pronouns or omitted subjects at the end of the text unit preceding the noise-removed unit;

[0070] Extract latent referential entities from forward-looking, non-noise text units;

[0071] Based on entity co-occurrence frequency and syntactic dependency relationship, the most likely referent is selected for explicit replacement;

[0072] Logical relationship analysis is performed on the conjunctions in the sentences before and after the removed position. If there is a break in the causal, adversative, or progressive relationship, standardized cohesive phrases are inserted to restore the coherence of the discourse.

[0073] As one embodiment of the present invention, the word embedding model in the financial field is a continuous bag-of-words model trained on a corpus of 1 billion financial news articles, with a vector dimension of 300, a training window size of 8, and a negative sampling number of 15.

[0074] As one embodiment of the present invention, the financial sentiment dictionary contains 3,200 sentiment terms, including 1,400 positive terms and 1,800 negative terms, with the sentiment intensity weight of each term ranging from 0.1 to 1.0.

[0075] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0076] 1. This invention introduces a dynamic calculation mechanism for the sentiment fluctuation index based on a sliding time window, abandoning the traditional fixed threshold filtering strategy. This allows the noise judgment threshold to be adaptively adjusted according to the intensity of market sentiment fluctuations, effectively addressing the noise surge caused by sudden rumors or panic messages.

[0077] 2. This invention employs a dual verification mechanism: first, candidate noise is initially screened using an emotion fluctuation index, and then a semantic stability assessment is used for secondary confirmation. This avoids the accidental deletion of effective high-emotion information (such as major policy announcements or financial report releases) due to reliance solely on emotion intensity. Furthermore, after noise removal, this invention performs contextual coherence restoration. Through referential resolution and logical connection reconstruction, it ensures the semantic integrity and fluency of the text sequence input to the large-scale financial model, significantly improving the stability and inference accuracy of subsequent model training. Experiments show that on a test set containing sudden social media rumors, compared to traditional preprocessing methods based on fixed thresholds, this invention improves the effective information retention rate by 27.4%, achieves a noise removal accuracy of 92.6%, and maintains a preprocessing throughput of over 2300 text units per second even in high-noise scenarios, meeting the performance requirements of real-time financial analysis. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the overall technical solution architecture of a text data preprocessing method suitable for large financial models proposed in this invention;

[0079] Figure 2 This is a schematic diagram of the core principle framework of the multi-level dynamic noise perception and adaptive filtering mechanism in this invention;

[0080] Figure 3 This is a flowchart illustrating the logical flow of the emotion fluctuation index calculation and dynamic threshold setting in this invention.

[0081] Figure 4 This is a logical flowchart of the semantic stability evaluation of candidate noise units in this invention;

[0082] Figure 5 This is a logical flowchart of the context coherence repair in this invention;

[0083] Figure 6 This is a schematic diagram of the multi-level interaction relationship and data flow between the real-time financial news stream, the preprocessing module, and the large financial model in this invention. Detailed Implementation

[0084] Please refer to Figures 1 to 6 This invention provides a text data preprocessing method suitable for large-scale financial models. Its core objective is to address the technical problems of sudden market sentiment noise in real-time financial news streams causing a sharp drop in preprocessing efficiency, and the inability of existing methods to dynamically adapt to sudden changes in noise intensity due to fixed threshold filtering, resulting in the accidental deletion of valid information. The method constructs a multi-level dynamic noise perception and adaptive filtering mechanism, combined with semantic stability assessment and contextual consistency verification, to achieve accurate identification and efficient removal of sudden sentiment noise in financial text streams, while retaining valid text fragments with real market signal value.

[0085] The method includes the following steps: receiving an original text sequence from a real-time financial news stream, the original text sequence consisting of multiple text units sorted by timestamp; performing initial word segmentation and part-of-speech tagging on the original text sequence to generate a lexical sequence with grammatical attributes; calculating the sentiment fluctuation index of each text unit based on the lexical sequence, the sentiment fluctuation index being determined by the deviation between the rate of change of sentiment word frequency density within a sliding time window and the historical baseline sentiment level; dynamically setting a sentiment noise judgment threshold for the current time window based on the sentiment fluctuation index, the sentiment noise judgment threshold being adaptively adjusted according to the standard deviation of the sentiment fluctuation index within the sliding time window; and setting a threshold for sentiment noise exceeding the threshold. The text units that meet the emotional noise threshold are labeled as candidate noise units. Semantic stability is evaluated on these candidate noise units by calculating the mean cosine similarity between the unit and its preceding and following non-noise units in the embedding space. If the mean cosine similarity is lower than a preset semantic consistency threshold, the candidate noise unit is identified as emotional noise and removed. Contextual coherence is restored on the text sequence after removing emotional noise, including dissolving referential inconsistencies and reconstructing logical connections at the text boundaries before and after the removed unit. The restored clean text sequence is output for subsequent training or inference using a large financial model.

[0086] In this step, the system receives a raw text sequence from a real-time financial news stream. This raw text sequence consists of multiple text units ordered by timestamps. The real-time financial news stream originates from authoritative financial media, stock exchange announcements, regulatory disclosures, and verified third-party news aggregation platforms. Each text unit is an independently published news message, announcement summary, or news flash, typically between 20 and 300 characters long, and includes a timestamp accurate to the millisecond. All text units are arranged in strict ascending order according to their timestamps, forming a continuous raw text sequence. This sequence serves as the input source for the entire preprocessing workflow; its completeness and temporal accuracy are prerequisites for ensuring the effectiveness of subsequent sentiment analysis. The system receives the raw text sequence through a high-throughput message queue, ensuring low latency and high reliability even under a load of processing over 2000 text units per second.

[0087] The original text sequence undergoes initial word segmentation and part-of-speech tagging to generate a lexical sequence with grammatical attributes. This step is performed by the natural language processing module integrated into the preprocessing engine front-end. This module employs a Chinese word segmentation algorithm based on conditional random fields, specifically optimized for financial terms such as "quantitative easing," "credit spread," and "short selling mechanism," ensuring that professional vocabulary is not incorrectly segmented. After segmentation, each lexical unit is assigned a part-of-speech tag, including 12 basic grammatical attributes such as nouns, verbs, adjectives, adverbs, numerals, and proper nouns. Furthermore, the system performs named entity recognition on financial entities such as company names, stock codes, and macroeconomic indicator names, and labels their types. The generated lexical sequence not only contains the original character form but also includes part-of-speech, entity type, and position index information, providing a structured foundation for subsequent sentiment keyword matching and syntactic dependency analysis.

[0088] Based on the lexical sequence, a sentiment fluctuation index is calculated for each text unit. This index reflects the degree of deviation of the emotional intensity carried by the current text unit from the recent market sentiment baseline, and is the core basis for determining whether it belongs to sudden noise. The calculation process first matches the emotional keywords in the current text unit from a preset financial sentiment dictionary. The financial sentiment dictionary contains 3200 sentiment terms, 1400 positive terms and 1800 negative terms, with the sentiment intensity weight of each term ranging from 0.1 to 1.0. The dictionary construction process includes collecting news texts published by authoritative financial media in the past 5 years with market reaction tags, using event-driven stock price fluctuations to infer the text's sentiment polarity, and manually verifying and labeling the intensity of frequently occurring emotional expressions. During matching, the system traverses all lexical units of the current text unit. If its character form is completely consistent with any sentiment term in the dictionary, the sentiment intensity weight corresponding to that term is extracted.

[0089] The raw sentiment score is obtained by summing the sentiment intensity weights of all sentiment keywords in the current text unit. If there are no sentiment keywords in the text unit, the raw sentiment score is 0. Subsequently, using a sliding time window of 30 text units, the moving average of the raw sentiment scores within the current window is calculated as the historical baseline sentiment level. The sliding time window advances one unit with the arrival of a new text unit, always maintaining a view of the most recent 30 processed text units. The historical baseline sentiment level represents the average sentiment state of the market in recent times, serving as a reference point for measuring the degree of current sentiment anomaly.

[0090] The absolute difference between the original sentiment score of the current text unit and the historical baseline sentiment level is calculated and divided by the historical baseline sentiment level to obtain the relative sentiment deviation. When the historical baseline sentiment level is 0, the relative sentiment deviation is directly taken as the absolute value of the original sentiment score to avoid division by zero error. Subsequently, the relative sentiment deviation is subjected to exponential smoothing to obtain the final sentiment fluctuation index. The exponential smoothing uses a first-order recursive formula:

[0091] ;

[0092] Indicates the current text unit The sentiment fluctuation index Representing text units The sentiment fluctuation index For its relative emotional bias, The smoothing coefficient is set to 0.3. This formula assigns higher weight to recent deviations while retaining historical trend information, effectively suppressing misjudgments caused by instantaneous spikes, and giving the sentiment fluctuation index time continuity and noise resistance.

[0093] Based on the aforementioned sentiment fluctuation index, a threshold for judging sentiment noise in the current time window is dynamically set. Traditional methods use a globally fixed threshold, which cannot meet the need to distinguish between normal high-sentiment information and noise during periods of intense market sentiment fluctuation. This invention abandons this static strategy and instead adaptively adjusts the threshold based on the dispersion of the sentiment fluctuation index within a sliding time window. Specifically, within the sliding time window, the sample standard deviation of the sentiment fluctuation index for all text units is calculated. The sample standard deviation reflects the overall variability of sentiment fluctuation within the current window; the larger the standard deviation, the more unstable the market sentiment and the higher the probability of noise. The sample standard deviation is multiplied by a preset sensitivity coefficient, which is set to 1.5. This coefficient was determined through extensive backtesting experiments to minimize the false deletion of effective information while ensuring a high noise detection rate. The product of the historical baseline sentiment level and the threshold is added to obtain the dynamic sentiment noise judgment threshold. This threshold changes in real time with the intensity of market sentiment fluctuations: during calm periods, the standard deviation is small, and the threshold is close to the historical baseline, only eliminating extreme outliers; during volatile periods, the standard deviation is large, and the threshold is significantly raised, avoiding misjudging reasonable emotional surges (such as major policy benefits) as noise.

[0094] Text units whose sentiment fluctuation index exceeds the aforementioned sentiment noise threshold are marked as candidate noise units. This step is a preliminary screening, based solely on the abnormality of sentiment intensity, and has not yet confirmed whether they are truly meaningless noise. Candidate noise units may contain two types of content: one is invalid noise such as social media rumors and repeated panic messages; the other is real but emotionally strong market signals, such as emergency interest rate cut announcements from central banks or sudden major losses disclosed by listed companies. Therefore, a second verification mechanism must be introduced to distinguish between the two.

[0095] The candidate noisy units are subjected to semantic stability evaluation. This evaluation aims to determine whether the candidate unit is semantically disconnected from its context, thereby confirming its noise attribute. Specifically, a pre-trained financial domain word embedding model is used to map the candidate noisy unit and its five preceding and five following non-noisy text units to a unified vector space. The financial domain word embedding model is a continuous bag-of-words model trained on a corpus of 1 billion financial news articles, with a vector dimension of 300, a training window size of 8, and a negative sampling rate of 15. This model fully captures the semantic relationships and contextual co-occurrence patterns among financial terms, ensuring that the vector representation has domain specificity.

[0096] For each non-noise text unit, average pooling is performed on its vector representation to obtain its context semantic vector. Average pooling generates a dense vector representing the overall semantics by calculating the arithmetic mean of all word vectors within the text unit. The cosine similarity between the semantic vector of the candidate noise unit and the semantic vectors of each context unit is calculated. Cosine similarity measures the directional proximity of two vectors, ranging from -1 to 1; a higher value indicates greater semantic similarity. The arithmetic mean of all cosine similarities is taken as the semantic stability index of the candidate noise unit. This index reflects the semantic coherence of the candidate unit with its surrounding valid information.

[0097] If the mean cosine similarity is lower than a preset semantic consistency threshold, the candidate noise unit is identified as emotional noise and removed. The preset semantic consistency threshold is 0.35. This threshold is determined by performing a grid search on a validation set containing labeled noise and valid high-emotion text, balancing recall and precision. When the semantic stability index is lower than this value, it indicates that the candidate unit has no substantial connection with the context and is likely an isolated rumor or a mechanically repetitive panic post, and should be removed; conversely, if the index is higher than this value, even if the emotional fluctuation is intense, it is retained as valid information due to its semantic coherence.

[0098] The text sequence after removing emotional noise undergoes contextual coherence restoration. Since the removal of noisy units may leave semantic breaks in the text stream, such as ambiguous referents or logical jumps, restoration is necessary to ensure the text quality input into the large financial model. The restoration process includes: identifying pronouns or omitted subjects at the end of the text unit preceding the removed noisy unit; the system parses sentence structure using a syntactic dependency analyzer to locate potential missing referents; and extracting potential referential entities from the preceding non-noisy text units. These entities include named entities such as company names, personal names, and economic indicators, sorted by their co-occurrence frequency in the preceding five text units. Based on entity co-occurrence frequency and syntactic dependency relationships, the most likely referent is selected for explicit replacement. For example, if "a certain technology company" is mentioned multiple times in the preceding text, and the sentence preceding the removed unit ends with "its stock price," then "its" is replaced with "the technology company."

[0099] Furthermore, the logical relationships between conjunctions in sentences before and after the removed position are analyzed. The system checks whether there are missing logical connections such as cause and effect, contrast, or progression at the beginning and end of the preceding and following text units. If a break in the relationship is detected, for example, if the previous sentence ends with "despite high inflation" and the next sentence begins with "the central bank maintained interest rates unchanged," with a missing transitional phrase in between, a standardized connecting phrase such as "however" is inserted to restore the coherence of the discourse. All repair operations are performed based on a combination of rules and statistical models to ensure that the repair results conform to the expression norms of financial texts.

[0100] The output is a clean, restored text sequence for use in subsequent training or inference of large-scale financial models. This sequence has been freed from sudden emotional noise while retaining accurate market signals and maintaining semantic coherence and integrity. The sequence is output to the input interface of downstream large-scale financial models in either standard text format or structured data stream form, supporting both batch processing and real-time streaming modes.

[0101] The above-described methods and steps constitute the core technical solution of this invention. At the system implementation level, the method is executed by a text data preprocessing system suitable for large-scale financial models. This system includes a raw text receiving module, a syntax parsing module, a sentiment fluctuation calculation module, a dynamic threshold setting module, a noise candidate labeling module, a semantic stability evaluation module, a noise confirmation and removal module, a context restoration module, and a clean text output module.

[0102] The raw text receiving module subscribes to data streams from financial news sources via a distributed message middleware, supporting multi-path concurrent access and flow control to ensure no data loss under high load. The syntax parsing module integrates a high-performance Chinese natural language processing engine, with a built-in financial domain word segmentation dictionary and named entity recognition model, capable of lexicalization and annotation of single text entries in milliseconds. The sentiment fluctuation calculation module maintains a circular buffer with a sliding time window, updating the raw sentiment score, historical baseline sentiment level, and sentiment fluctuation index in real time; the calculation process is highly optimized to support thousands of queries per second.

[0103] The dynamic threshold setting module continuously monitors the statistical distribution of the sentiment fluctuation index within the sliding window, calculates the sample standard deviation in real time, and generates a dynamic threshold. The noise candidate labeling module assigns candidate noise labels to each text unit based on the threshold comparison results and temporarily stores them in the evaluation queue. The semantic stability evaluation module calls a pre-loaded financial word embedding model to batch process candidate units and their contexts, calculating the mean cosine similarity. This module uses vectorized operations for acceleration, significantly improving evaluation efficiency.

[0104] The noise identification and removal module determines whether to physically delete candidate units from the text sequence based on a comparison of semantic stability indicators with a preset lower limit. The context repair module is triggered immediately after the deletion operation, performing referential resolution and logical connection reconstruction. Internally, it includes a financial entity knowledge base and a linking phrase template library to ensure the professionalism and naturalness of the repair results. The clean text output module encapsulates the repaired text sequence into a standard data package and pushes it to the training or inference pipeline of the large financial model via a high-speed network interface.

[0105] The entire system adopts a microservice architecture, with modules interacting via lightweight communication protocols, supporting horizontal scaling and fault tolerance. Deployed on a low-latency computing cluster, the system meets the stringent timeliness requirements of real-time financial analysis. Experimental results show that, on a test set containing sudden social media rumors, this invention, compared to traditional preprocessing methods based on fixed thresholds, achieves a 27.4% improvement in effective information retention, a noise removal accuracy of 92.6%, and maintains a preprocessing throughput of over 2300 text units per second even in high-noise scenarios.

[0106] In summary, this invention resolves the contradiction between noise adaptability and information preservation in financial text preprocessing through a dynamic threshold mechanism and semantic stability dual verification, providing high-quality and highly coherent input data for large financial models, and significantly improving the robustness and prediction accuracy of the models in real market environments.

Claims

1. A text data preprocessing method suitable for large-scale financial models, characterized in that, include: Receive raw text sequences from a real-time financial news stream, the raw text sequences consisting of multiple text units sorted by timestamp; The original text sequence is initially segmented and part-of-speech tagged to generate a word sequence with grammatical attributes; Based on the lexical sequence, the sentiment fluctuation index of each text unit is calculated, including: The system matches emotional keywords in the current text unit from a pre-defined financial sentiment dictionary, which includes a positive sentiment word set and a negative sentiment word set, with each type of word labeled with an emotional intensity weight. The raw emotion score is obtained by summing the emotion intensity weights of all emotion keywords in the current text unit. Using a sliding time window of 30 text units, the moving average of the original sentiment score within the current window is calculated as the historical baseline sentiment level. Calculate the absolute difference between the original sentiment score of the current text unit and the historical baseline sentiment level, and divide it by the historical baseline sentiment level to obtain the relative sentiment deviation. The relative emotional deviation is subjected to exponential smoothing to obtain the final emotional fluctuation index. The emotion fluctuation index is determined by the deviation between the rate of change of emotion word frequency density within a sliding time window and the historical baseline emotion level. Based on the emotional fluctuation index, the emotional noise judgment threshold for the current time window is dynamically set, and the emotional noise judgment threshold is adaptively adjusted according to the standard deviation of the emotional fluctuation index within the sliding time window. Text units whose emotional fluctuation index exceeds the emotional noise determination threshold are marked as candidate noise units; The semantic stability of the candidate noise unit is evaluated by calculating the mean cosine similarity between the unit and its adjacent non-noise units in the embedding space. If the mean cosine similarity is lower than the preset semantic consistency lower limit, the candidate noise unit is identified as emotional noise and is removed. The text sequence after removing emotional noise is subjected to contextual coherence restoration, which includes the resolution of references and the reconstruction of logical connections at the text boundaries before and after the removed unit. Output the repaired clean text sequence for use in subsequent training or inference of large financial models.

2. The text data preprocessing method applicable to large financial models according to claim 1, characterized in that, Dynamically set the threshold for judging emotional noise in the current time window, including: Within the sliding time window, calculate the sample standard deviation of the sentiment fluctuation index for all text units; Multiply the sample standard deviation by a preset sensitivity coefficient, wherein the sensitivity coefficient is 1.5; The dynamic emotion noise judgment threshold is obtained by adding the product of the historical baseline emotion level and the sample standard deviation multiplied by a preset sensitivity coefficient.

3. The text data preprocessing method for large financial models according to claim 2, characterized in that, The semantic stability evaluation of the candidate noise units includes: Using a pre-trained word embedding model in the financial domain, candidate noisy units and their five forward and five backward non-noisy text units are mapped to a unified vector space. Average pooling is performed on the vector representation of each non-noise text unit to obtain its context semantic vector; Calculate the cosine similarity between the semantic vector of the candidate noise unit and the semantic vector of each context; The arithmetic mean of all cosine similarities is taken as the semantic stability index of the candidate noise unit.

4. The text data preprocessing method for large financial models according to claim 3, characterized in that, The preset semantic consistency lower limit is 0.

35.

5. The text data preprocessing method for large financial models according to claim 4, characterized in that, The financial word embedding model is a continuous bag-of-words model trained on a corpus of 1 billion financial news articles. Its vector dimension is 300, the training window size is 8, and the number of negative samples is 15.

6. The text data preprocessing method for large financial models according to claim 5, characterized in that, The financial sentiment dictionary is constructed in the following way: Collect news texts published by authoritative financial media in the past 5 years with accompanying market reaction tags; Inferring the polarity of textual sentiment by using the magnitude of event-driven stock price fluctuations; Frequently occurring emotional expressions are manually verified and their intensity is calibrated. A structured dictionary containing 3,200 emotion terms was created, including 1,400 positive terms and 1,800 negative terms. The emotion intensity weight of each term ranges from 0.1 to 1.

0.

7. The text data preprocessing method for large financial models according to claim 6, characterized in that, The context coherence restoration includes: Identify pronouns or omitted subjects at the end of the text unit preceding the noise-removed unit; Extract latent referential entities from forward-looking, non-noise text units; Based on entity co-occurrence frequency and syntactic dependency relationship, the most likely referent is selected for explicit replacement; Logical relationship analysis is performed on the conjunctions in the sentences before and after the removed position. If there is a break in the causal, adversative, or progressive relationship, standardized cohesive phrases are inserted to restore the coherence of the discourse.

8. The text data preprocessing method for large financial models according to claim 7, characterized in that, The original text sequence is derived from authoritative financial media, stock exchange announcements, regulatory agency disclosures, or third-party news aggregation platforms with verified credibility. Each text unit is between 20 and 300 characters long and includes a millisecond-level timestamp.

9. The text data preprocessing method for large financial models according to claim 8, characterized in that, The initial word segmentation uses a Chinese word segmentation algorithm based on conditional random fields, and is specifically optimized for financial terms. At the same time, named entity recognition is performed to mark company names, stock codes and macroeconomic indicator names.

Images