Method and device for training financial public opinion analysis model, and computer device

Abstract

The application relates to the technical field of model training, and discloses a training method and device of a financial public opinion analysis model and a computer device. The method comprises the following steps: obtaining unstructured text data of a financial social platform and a screenshot uploaded by a user; performing semantic processing on the text data to extract a text emotion time sequence feature, and performing visual feature extraction on the screenshot to obtain an image trend time sequence feature; performing spatiotemporal alignment processing on the features to obtain an aligned text-image feature pair; obtaining the original text and the screenshot after time alignment as the aligned text-image data according to a time mapping relationship; performing multi-modal deep comparison on the data to generate a structured comparison label; constructing a training set based on the aligned text-image feature pair and the structured comparison label; and training a preset model by using the training set to obtain a final financial public opinion analysis model. The application can improve the training efficiency and accuracy of the model.

Description

Technical Field

[0001] This application relates to the field of model training technology, and in particular to a training method, apparatus, and computer equipment for a financial public opinion analysis model. Background Technology

[0002] Financial sentiment data often appears in the form of both text and images. Analyzing such multimodal data requires addressing the effective representation and fusion of cross-modal features. Current technologies for processing this type of multimodal data typically extract features from each modality separately and then directly concatenate or simply weight them, lacking methods for modeling the deep semantic interactions between text and image features, resulting in insufficient feature representation capabilities. Furthermore, existing model training schemes are relatively simplistic in their loss function construction and parameter optimization strategies, easily leading to training instability or low convergence efficiency when faced with multimodal inputs, making it difficult to obtain reliable analytical models. Summary of the Invention

[0003] Based on this, it is necessary to address the technical problems of instability and low convergence efficiency in the training and analysis models of existing technologies, and propose a training method for financial public opinion analysis models.

[0004] Firstly, a training method for a financial public opinion analysis model is provided, the method comprising: Obtain unstructured text data and user-uploaded screenshots from financial social platforms; Semantic processing is performed on the unstructured text data to extract text sentiment time series features; Visual features are extracted from the screenshot to obtain image trend time series features; The text sentiment time series features and the image trend time series features are spatiotemporally aligned to obtain aligned text-image feature pairs. Based on the time mapping relationship in the aligned text-image feature pairs, the original text and screenshot after time alignment are obtained as aligned text-image data. Multimodal depth comparison is performed on the aligned text and image data to generate structured comparison labels; A training set is constructed based on the aligned text-image feature pairs and the structured contrast labels. The pre-set financial public opinion analysis model is trained using the training set to obtain the trained financial public opinion analysis model.

[0005] Secondly, a training device for a financial public opinion analysis model is provided, the device comprising: The acquisition module is used to acquire unstructured text data and screenshots uploaded by users from financial social platforms; The extraction module is used to perform semantic processing on the unstructured text data and extract the text sentiment time series features; The extraction module is also used to extract visual features from the screenshot to obtain image trend time series features; The alignment module is used to perform spatiotemporal alignment processing on the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs. The generation module is used to obtain the original text and screenshot after time alignment based on the time mapping relationship in the aligned text-image feature pair, as aligned text-image data; The generation module is also used to perform multimodal depth comparison on the aligned text and image data to generate structured comparison labels; The construction module is used to construct a training set based on the aligned text-image feature pairs and the structured contrast labels; The training module is used to train the preset financial public opinion analysis model using the training set to obtain the trained financial public opinion analysis model.

[0006] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for the above-mentioned financial public opinion analysis model.

[0007] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the training method for the aforementioned financial public opinion analysis model.

[0008] The beneficial effects of this application are: This application extracts text sentiment time series features and image trend time series features separately, and aligns them spatiotemporally, so that the features of the text modality and the image modality are established in a corresponding relationship in the time dimension. Compared with the existing technology that directly splices or simply weights the features of each modality, this application realizes cross-modal semantic alignment and association modeling at the feature level, which can more fully capture the deep semantic interaction information between text content and image content, thereby improving the representation ability of multimodal features.

[0009] This application obtains the original text and screenshot after time alignment based on the time mapping relationship in the aligned text-image feature pairs, so that the subsequent multimodal deep comparison operation is based on the time-consistent text-image data, avoiding the comparison deviation caused by time misalignment, and ensuring the accuracy and reliability of the comparison label generation.

[0010] This application generates structured contrast labels by performing multimodal deep comparison on time-aligned text and image data, which quantifies the semantic consistency between text and images in a structured form, providing clear supervision signals for subsequent model training. This ensures the quality of the labels in the training set and helps the model establish accurate feature-label mapping relationships during training.

[0011] This application constructs a training set based on aligned text-image feature pairs and structured comparative labels, so that the training data simultaneously covers aligned multimodal feature representations and corresponding supervision label information. The data organization method of the training set matches the model input requirements, which is conducive to improving the stability and convergence efficiency of model training, thereby obtaining a reliable financial sentiment analysis model. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] in: Figure 1 This is a schematic diagram of the system architecture in one embodiment; Figure 2 This is a flowchart of a training method for a financial public opinion analysis model in one embodiment; Figure 3 This is a structural block diagram of a training device for a financial public opinion analysis model in one embodiment; Figure 4 This is a schematic diagram of a computer storage medium in one embodiment; Figure 5 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0014] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0015] like Figure 1As shown, the model training system provided in this application embodiment includes a user terminal 100 and a computer device 101. The user terminal 100 and the computer device 101 establish a data transmission connection through a communication network. The communication network can be any one or a combination of wired network and wireless network.

[0016] User terminal 100 is an electronic computing device with data input and display output capabilities, and its specific form can be a desktop computer, laptop computer, tablet computer, or smartphone. User terminal 100 is used by operators to initiate model training commands, configure training parameters, upload raw text and image data, and view training results. Operators send training start commands and training parameter configuration information to computer device 101 through user terminal 100. Training parameters include, but are not limited to, training batch size, initial learning rate, maximum number of training epochs, early stopping patience value, and convergence threshold. After training is completed, user terminal 100 receives and displays the training results, including the loss function curve, classification accuracy index, F1 score for each class, and the final model evaluation report.

[0017] Computer device 101 is a server or workstation with large-scale parallel computing capabilities, used to carry out training and inference operations for the financial public opinion analysis model. Computer device 101 receives training parameters and training data sent by user terminal 100, executes the model training method of this application, including the collection and preprocessing of raw text and image data, the construction of multimodal input features, the organization of the training set, and iterative training of the financial public opinion analysis model. After training is completed, computer device 101 serializes and stores the trained financial public opinion analysis model, deploys it to the inference service module, receives newly collected text and image data, preprocesses it to generate multimodal input features, and outputs public opinion consistency classification results.

[0018] User terminal 100 and computer device 101 form a client-server architecture. User terminal 100, as the client side, handles human-computer interaction and command issuance, while computer device 101, as the server side, handles data storage, model training, and inference computation. After the operator initiates the training task and configures the parameters on user terminal 100, computer device 101 autonomously executes all training operations. During training, computer device 101 periodically pushes training progress and intermediate metrics to user terminal 100 for operator monitoring. After training is completed, computer device 101 sends the final model and evaluation results back to user terminal 100 for archiving, while simultaneously retaining complete training records locally for subsequent auditing and model iteration.

[0019] It should be understood that Figure 1 The number and structure of the various components of the computer equipment and user terminals shown are for illustrative purposes only. The appropriate number of components and structures can be implemented according to specific needs.

[0020] Please see Figure 2 As shown, Figure 2 A flowchart illustrating the training method for the financial public opinion analysis model provided in this application embodiment includes the following steps: S1. Obtain unstructured text data and screenshots uploaded by users from financial social platforms.

[0021] The computer equipment establishes a communication connection with the data interface of the financial social platform via a network interface, and extracts unstructured text data from the public information stream of the financial social platform according to a preset data collection strategy. Unstructured text data includes, but is not limited to, the text of user-posted posts, comments, discussion threads, and publicly available portions of private messages. Simultaneously, the computer equipment collects screenshots uploaded by users that are associated with the aforementioned unstructured text data in terms of posting time. These screenshots include user-captured stock quotes, fund net asset value trends, account profit and loss screenshots, and summaries of financial news. The computer equipment assigns a unique data identifier to each of the collected unstructured text data and screenshots, and establishes an initial association between them using these identifiers to form raw text-image data pairs. These raw text-image data pairs are then stored in a temporary cache in the local database for subsequent processing. During the data collection process, the computer equipment performs deduplication filtering on duplicate data and marks and removes data lacking key fields to ensure that the data entering subsequent processes is complete and free of redundancy.

[0022] For example, a computer device retrieves a user post from the stock discussion section of a financial forum. The post's body contains the user's subjective judgment on the short-term trend of a certain stock. The post also includes three screenshots: a chart of the stock's intraday price movement, a screenshot of the user's personal account's profit and loss, and a screenshot of a financial news headline. The computer device assigns the post's body the ID TXT-20240601-001, and the three screenshots the IDs IMG-20240601-001a, IMG-20240601-001b, and IMG-20240601-001c. It then establishes an initial association between TXT-20240601-001 and the three screenshot IDs, storing them in a temporary cache.

[0023] S2. Perform semantic processing on unstructured text data to extract text sentiment time series features.

[0024] The computer device performs semantic processing on unstructured text data in the cache. First, using Retrieval Enhancement Generation (RAG) technology, the device retrieves semantically relevant contextual data from an external financial knowledge base, using keywords from the unstructured text data as query vectors. This contextual data includes historical announcements of the financial instruments mentioned in the text, industry research report summaries, and relevant policy interpretations. Based on the retrieved contextual data, the computer device generates a semantic enhancement vector. This vector is obtained by encoding and aggregating the contextual data, serving as auxiliary information for subsequent semantic understanding.

[0025] Secondly, the computer device performs high-frequency new word monitoring on unstructured text data. By statistically analyzing word frequencies and comparing them with existing financial ontology knowledge bases, when an unknown concept not covered by the existing financial ontology is identified, the computer device extracts its semantic definition from the context of the current text and uses this definition as the current contextual definition of the unknown concept. The computer device dynamically writes this current contextual definition into the dynamic prompt word space of the large language model, enabling the pre-set financial ontology semantics in the large language model to evolve and correct according to the new concept. This allows the large language model to understand and process emerging financial terms that were originally beyond its knowledge scope.

[0026] Subsequently, the computer device, combining semantic enhancement vectors, utilizes an evolutionarily corrected large language model to perform semantic understanding on the unstructured text data, generating enhanced semantic information. This enhanced semantic information includes at least ontology-corrected semantic vectors, entity tags, and sentiment tags. Entity tags identify the names of financial entities involved in the text, while sentiment tags identify the polarity and intensity of the emotions expressed in the text. Finally, based on the sentiment tags in the enhanced semantic information, the computer device arranges the unstructured text data according to its publication time sequence and extracts the text sentiment time-series features. These features form a sequence vector with timestamps on the horizontal axis and sentiment polarity intensity values ​​on the vertical axis.

[0027] For example, when processing a post containing the non-standard term "quantitative sell-off," the computer device, lacking an entry for this term in the existing financial ontology, identifies from the context that the term refers to the behavior of concentrated selling through algorithmic trading strategies leading to a drop in stock prices. This definition is then written into the dynamic cue word space of the large language model for evolutionary correction. After semantic understanding, the corrected large language model outputs enhanced semantic information, with a semantic vector dimension of 768. Entity labels include a quantitative private equity fund and the CSI 300 index, and the sentiment label is bearish with an intensity of 0.82. The computer device then arranges this sentiment label chronologically to form a text sentiment time series feature, which is represented by a sentiment intensity sequence sampled every 5 minutes from the time of publication.

[0028] S3. Extract visual features from the screenshot to obtain image trend time series features.

[0029] The computer device retrieves user-uploaded screenshots associated with unstructured text data from a temporary cache and performs visual feature extraction on each screenshot sequentially. The computer device uses a pre-trained convolutional neural network model to encode the screenshots, mapping the pixel information of the screenshots into high-dimensional feature vectors. These feature vectors represent the visual semantic content contained in the screenshots, such as candlestick chart patterns, numerical annotations, color distribution, and text region information.

[0030] Based on this, the computer equipment performs trend recognition on the core visual elements in the screenshot, specifically detecting and quantifying trend-related visual elements such as price curves, price fluctuation indicators, and volume bar charts. The computer equipment sorts the detection results according to the screenshot's capture or extraction time, generating an image trend time series feature indexed by timestamps and containing trend quantification values ​​as elements. Trend quantification values ​​include, but are not limited to, the slope direction of the price curve shown in the screenshot, the percentage fluctuation range, and numerical information about key support and resistance levels. The computer equipment stores the image trend time series feature as a fixed-length vector sequence for subsequent alignment and computation with text sentiment time series features.

[0031] For example, a computer device extracts visual features from a screenshot showing the intraday price movement of a stock. The convolutional neural network model outputs a 1024-dimensional feature vector, which includes trend recognition results indicating that the screenshot shows an upward trend with a positive slope and a maximum increase of 3.5%. The computer device records the screenshot as being taken at 14:30 on June 1, 2024, and incorporates this trend quantification value into the image trend time series features. The vector element corresponding to 14:30 encodes an upward trend with an intensity of 0.75.

[0032] S4. Perform spatiotemporal alignment processing on the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs.

[0033] The computer device sets the text sentiment time series features obtained in step S2 as the reference sequence and the image trend time series features obtained in step S3 as the sequence to be aligned. Since the release time and capture time of the text data and screenshot data may not be completely synchronized during the actual acquisition process, and the two sequences are offset on the time axis, the computer device uses a dynamic time warping algorithm to flexibly align the two sequences.

[0034] The Dynamic Time Warping algorithm constructs a distance matrix between a reference sequence and the sequence to be aligned. It then uses dynamic programming to find the path with the minimum cumulative distance between the two sequences, reflecting their optimal mapping relationship in the time dimension. The computer calculates the time offset of the sequence to be aligned relative to the reference sequence based on this minimum distance path, and performs time correction on the image trend time series features according to this offset, aligning the corrected image trend time series features with the text sentiment time series features on the time axis.

[0035] The computer equipment pairs the two corrected sequences one by one according to time points, forming aligned text-image feature pairs. Each text-image feature pair consists of a text sentiment feature vector and an image trend feature vector at the same time point. The aligned text-image feature pairs contain a time mapping relationship, which records the actual time point corresponding to each pair.

[0036] For example, the text sentiment time series feature has 6 sampling points between 10:00 and 11:00, and the image trend time series feature has 6 sampling points between 9:55 and 10:55, with a system offset of about 5 minutes between them. The dynamic time warping algorithm calculates that the minimum distance path indicates that the overall image sequence is lags by 5 minutes. Based on this, the computer moves each time point of the image sequence forward by 5 minutes. After correction, the two are paired at time points such as 10:00, 10:10, and 10:20, resulting in 6 sets of aligned text-image feature pairs. Each feature pair contains both the text sentiment vector and the image trend vector at that time point.

[0037] S5. Based on the time mapping relationship in the aligned text and image feature pairs, obtain the original text and screenshot after time alignment, and use them as aligned text and image data.

[0038] The computer device reads the time mapping relationship recorded in the aligned text-image feature pairs. This time mapping relationship is stored in the form of a table mapping data identifiers to corrected time points. Based on this table, the computer device retrieves the original unstructured text data and original screenshot data that match each corrected time point from a temporary cache. The retrieved original text and screenshots are the data that has been aligned in the time dimension, and the computer device uniformly marks them as aligned text-image data.

[0039] In the aligned text and image data, each set of data consists of a segment of original text and one or more original screenshots at the same corrected time point, with strict temporal semantic correspondence between the two. The computer equipment transfers the aligned text and image data from a temporary buffer to a dedicated aligned data storage area and clears the aligned original data from the temporary buffer to free up storage space. The purpose of this step is to provide raw data input that is strictly synchronized in the time dimension for subsequent multimodal depth comparison, avoiding comparison bias caused by temporal misalignment.

[0040] For example, based on the time mapping relationship obtained in step S4, the computer device backtracks to retrieve the original post text with the ID TXT-20240601-001 and the original intraday trend screenshot with the ID IMG-20240601-001a from the pairing item corresponding to 10:00, and combines these two into a set of aligned text and image data. Similarly, at the time point of 10:10, another combination of text and screenshot is obtained, ultimately forming a data set containing multiple sets of aligned text and image data.

[0041] S6. Perform multimodal depth comparison on the aligned text and image data to generate structured comparison labels.

[0042] The computer device inputs the aligned text and image data into a pre-trained multimodal large-scale model group by group. This model is capable of simultaneously processing text and image inputs, and can perform joint understanding and cross-reasoning on information from both modalities. The computer device uses the multimodal large-scale model to extract investor subjective statements from the text portion of the aligned text and image data. These statements include investors' predictions of the future trend of financial instruments, their subjective evaluation of market sentiment, and their intentions regarding their own investment behavior. Simultaneously, the computer device uses the multimodal large-scale model to extract objective profit and loss facts from the image portion of the aligned text and image data. These facts include quantifiable objective information such as the actual price changes shown in the screenshot, realized profits and losses, and total account assets.

[0043] Subsequently, the computer system cross-references investors' subjective statements with objective profit and loss facts to determine the degree of consistency between the two. The degree of consistency is categorized into three levels: highly consistent, partially consistent, and obviously contradictory. Based on the comparison results, the computer system generates structured comparison labels. These labels record the consistency level and the corresponding specific comparison criteria using standardized classification codes. For example, a subjective view of optimism coupled with objective losses is labeled as contradictory and includes the specific numerical differences. These structured comparison labels are used as supervisory signals for subsequent training samples.

[0044] For example, the multimodal large model extracts from text an investor's subjective opinion that they are strongly bullish on the stock's continued rise the next day, and from the screenshot, extracts the objective profit and loss facts that the stock fell by 2.3% that day, resulting in a floating loss of 15,000 yuan. After comparison by computer equipment, it is determined that the subjective opinion and the objective facts clearly contradict each other, generating a structured comparison label, coded as CLASS-CONTRADICT-001, with the annotation stating that the subjective bullish opinion contradicts the 2.3% drop that day and the floating loss of 15,000 yuan.

[0045] S7. Construct a training set based on aligned text-image feature pairs and structured contrast labels.

[0046] The computer device extracts the text sentiment time-series feature vector and image trend time-series feature vector for each pair of aligned text-image feature pairs. These two vectors are then combined into a single multimodal input feature through feature concatenation or feature fusion. The dimension of this multimodal input feature is the sum of the dimensions of the two single-modal features or a unified dimension after dimensionality reduction. The computer device converts the structured contrast labels into classification labels, which are then encoded using one-hot encoding or integer encoding, serving as supervision labels for the corresponding training samples.

[0047] The computer device uses each set of multimodal input features and its corresponding supervision label to form a training sample. All training samples are then aggregated and divided into a training subset and a validation subset according to a preset ratio. The training subset is used for model parameter learning, and the validation subset is used for performance monitoring during training. The computer device randomly shuffles the samples in the training subset to eliminate the influence of sample order on training, ultimately forming a complete training set and storing it in the training data storage area.

[0048] For example, a set of aligned text-image feature pairs contains a 768-dimensional text sentiment feature vector and a 1024-dimensional image trend feature vector. The computer concatenates these two into a 1792-dimensional multimodal input feature. The corresponding structured contrast label CLASS-CONTRADICT-001 is converted into integer-coded label 2, and this set of data constitutes a training sample. The computer processes all 5000 sets of aligned text-image data collected that day to obtain 5000 training samples, which are then divided into 4000 training samples and 1000 validation samples in an 8:2 ratio to form the training set.

[0049] S8. Use the training set to train the pre-set financial public opinion analysis model to obtain the trained financial public opinion analysis model.

[0050] The computer device inputs the multimodal input features from the training set constructed in step S7 into a pre-defined financial public opinion analysis model in batches. This financial public opinion analysis model can be a Large Language Model (LLM). In this embodiment, the LLM has a multimodal input interface, capable of receiving concatenated multimodal input features and performing forward inference calculations. After performing internal attention mechanism operations and hierarchical feature transformations on the input multimodal input features, the LLM outputs predicted labels, which are the model's determination of the consistency category to which the current input sample belongs.

[0051] The computer device calculates the loss function value between the predicted label and the corresponding supervised label in the training set. The loss function is the cross-entropy loss function, which is calculated as L = -1 divided by N multiplied by the sum of terms from i = 1 to N, where yi is the one-hot encoding vector of the supervised label of the i-th sample, yi_hat is the probability distribution vector of the predicted label of the i-th sample, and N is the number of samples in the current batch. The computer device optimizes by minimizing the loss function value, using the backpropagation algorithm to calculate the gradient and iteratively update the model parameters of the Large Language Model (LLM). The learning rate is gradually reduced in each iteration according to a preset decay strategy.

[0052] After each training epoch, the computer calculates the validation accuracy using a validation subset. When the validation accuracy no longer improves after several consecutive epochs and the loss function value falls below a preset convergence threshold, the computer determines that training has converged and stops iterating. It then outputs the Large Language Model (LLM) with the current parameter state as the completed financial sentiment analysis model. The completed financial sentiment analysis model can determine the consistency of newly input financial text and image data, outputting a classification result showing the consistency between investors' subjective statements and objective facts.

[0053] For example, a computer device divides 4000 training samples into mini-batches of 64 samples each and feeds them into a Large Language Model (LLM). The initial learning rate is set to 0.001, and the learning rate is decayed by multiplying it by 0.9 every 10 rounds. After 50 rounds of training, the validation set accuracy stabilizes at 92.5%, and the loss function value drops below 0.15, which is below the preset convergence threshold of 0.2. At this point, the computer device stops training and outputs the trained Large Language Model (LLM). When this model receives a new set of financial text and image data, it can automatically output highly consistent, partially consistent, or obviously contradictory classification results.

[0054] In one possible embodiment, S2, the semantic processing steps, include: S21. Use RAG technology to retrieve the associated contextual data of the unstructured text data, and generate semantic enhancement vectors based on the associated contextual data as auxiliary information for semantic understanding.

[0055] The computer device executes the Retrieval Enhancement Generation (RAG) technique on each line of unstructured text data obtained in step S1. Specifically, the computer device first performs word segmentation and semantic encoding on the unstructured text data to be processed, transforming the text content into a high-dimensional query vector that represents the core semantic information of the text. The computer device then performs similarity matching calculations between this query vector and a pre-built external financial knowledge base. This external financial knowledge base stores a large amount of contextual data related to the financial field, including historical announcements of financial targets, industry research report summaries, regulatory policy interpretation documents, and in-depth analysis articles from professional financial media.

[0056] The computer device employs a vector cosine similarity calculation method to select several relevant contextual data points from an external financial knowledge base whose similarity to the current query vector exceeds a preset threshold, set between 0.75 and 0.85. The device then sorts the selected relevant contextual data points from highest to lowest similarity score, selecting the top K points as the search results, where K is set to 5 to 10. Subsequently, the device encodes and aggregates these K relevant contextual data points, generating a semantic enhancement vector by performing a weighted summation operation on the semantic vectors of each relevant contextual data point. The weighting coefficients are proportionally distributed based on the similarity score between each relevant contextual data point and the current unstructured text data. This semantic enhancement vector serves as auxiliary information in the subsequent semantic understanding process, supplementing the background knowledge and domain context that may be missing from the current unstructured text data itself.

[0057] For example, when processing a post containing information about a stock experiencing a surge in trading volume due to unusual activity in major funds, the computer device encodes the text into a 768-dimensional query vector and matches it with an external financial knowledge base. The matching result returns eight related contextual data points. The three with the highest similarity are a summary of the stock's recent 30-day trading data, an analysis report on fund flows in the same industry sector, and a policy-favorable announcement regarding the stock's related concept sector. The computer device then weights and sums the semantic vectors of these three related contextual data points according to similarity weights of 0.5, 0.3, and 0.2, generating a one-dimensional 768-dimensional semantic enhancement vector. This vector contains supplementary semantic information such as the movement of major funds, sector linkage effects, and policy support.

[0058] S22. Detect high-frequency new words in the unstructured text data. When an unknown concept not covered in the existing financial ontology is identified, obtain the current contextual definition of the unknown concept.

[0059] After completing the S21 retrieval process, the computer device performs high-frequency new word monitoring on the unstructured text data. First, the computer device performs word segmentation on the unstructured text data, obtaining a word sequence composed of several words, and then counts the frequency of each word in the current batch of all unstructured text data. The computer device then compares the obtained word frequency data with a pre-built financial ontology knowledge base, which includes known standard terms, professional concepts, and their hierarchical relationships in the financial field.

[0060] When a computer detects that a word appears more frequently than a preset high-frequency threshold and that the word does not exist in the financial ontology knowledge base, the computer marks the word as an unknown concept. The high-frequency threshold is set to appear at least three times in the current batch of data. For a marked unknown concept, the computer locates the original sentence containing the unknown concept from the unstructured text data, extracts several words before and after the unknown concept as context windows, and sets the length of the context windows to 5 to 10 words before and after the unknown concept. The computer performs semantic analysis on the words in the context windows, inferring the specific meaning of the unknown concept in the current context, and uses the inferred meaning as the current contextual definition of the unknown concept.

[0061] For example, when processing a batch of unstructured text data containing 2,000 posts, the computer device found that the term "Beijing Stock Exchange IPO arbitrage" appeared 7 times in the data for that day. However, the financial ontology knowledge base only includes three independent entries for "Beijing Stock Exchange IPO arbitrage," not the combined concept of "Beijing Stock Exchange IPO arbitrage." The computer device marked "Beijing Stock Exchange IPO arbitrage" as an unknown concept and extracted a context window from the statements containing the term. From the statement "Recently, the arbitrage space for Beijing Stock Exchange IPOs has narrowed, and the winning rate has dropped significantly," it inferred that the concept refers to an investment strategy of subscribing to new shares on the Beijing Stock Exchange and profiting from the price difference between the primary and secondary markets. The computer device recorded this inferred meaning as the current contextual definition of "Beijing Stock Exchange IPO arbitrage."

[0062] S23. Dynamically update the current context definition to the dynamic prompt word space of the large language model to perform evolutionary correction on the financial ontology semantics in the large language model.

[0063] The computer device writes the current contextual definition of the unknown concept obtained in step S22 into the dynamic cue word space of the large language model. The dynamic cue word space is a cue word storage area that the computer device pre-allocates for the large language model and can modify in real time. The cue word content in this area is loaded into the input terminal before the large language model performs each inference task to guide the understanding direction of the large language model.

[0064] Specifically, the computer device organizes the current context definition into a prompt word instruction according to a preset format template. This instruction includes the name of the unknown concept, a detailed description of the current context definition, and an explanation of the relationship between this concept and related concepts already existing in the financial ontology knowledge base. The computer device appends this prompt word instruction to the end of the existing set of financial ontology prompt words in the dynamic prompt word space, thereby expanding the semantic content of the financial ontology in the dynamic prompt word space.

[0065] When the large language model performs subsequent semantic understanding tasks, the computer loads all the prompt words in the updated dynamic prompt word space as system-level instructions into the input of the large language model. Guided by these instructions, the large language model expands its understanding of financial concepts, incorporating previously uncovered unknown concepts into its semantic understanding system, thereby achieving evolutionary correction of the financial ontology semantics within the large language model. After each round of semantic processing, the computer deduplicates and reorganizes the prompt words in the dynamic prompt word space to prevent the same concept from being repeatedly written, which could lead to the expansion of the prompt word space.

[0066] For example, the computer device organizes the current contextual definition of "Beijing Stock Exchange IPO arbitrage" into a prompt word instruction concept, defined as an investment strategy of subscribing to new shares on the Beijing Stock Exchange and profiting from the price difference between the primary and secondary markets. This is related to the existing concepts of "Beijing Stock Exchange IPO subscription" and "A-share IPO subscription," but applies to different markets. This instruction is then added to the dynamic prompt word space. Subsequently, when the large language model receives new text containing "Beijing Stock Exchange IPO arbitrage," it can accurately understand the meaning of the concept based on the definition already written in the dynamic prompt word space, instead of breaking it down into three independent words for separate understanding, thus avoiding the problem of semantic fragmentation.

[0067] S24. Combining the semantic enhancement vector, the unstructured text data is semantically understood using the evolution-corrected large language model to generate enhanced semantic information.

[0068] The computer device combines the semantic enhancement vector generated in step S21 with the large language model after evolution correction in step S23 to perform semantic understanding on unstructured text data. The computer device first fuses the semantic enhancement vector with the original semantic encoding of the unstructured text data through vector concatenation to form a composite input representation carrying supplementary background knowledge.

[0069] Subsequently, the computer device inputs the composite input representation along with the set of financial ontology prompts loaded from the dynamic prompt space into the large language model. Guided by the set of financial ontology prompts, the large language model, combined with the contextual information carried by the semantic enhancement vectors in the composite input representation, performs deep semantic parsing of the unstructured text data. The parsing process of the large language model includes entity recognition, relation extraction, and sentiment determination for each semantic unit in the text, outputting structured semantic parsing results.

[0070] The computer system processes the parsing results of the large language model to generate enhanced semantic information. This enhanced semantic information comprises at least three components: semantic vectors corrected by ontology evolution, entity labels, and sentiment labels. The semantic vectors corrected by ontology evolution represent the overall semantic meaning of the unstructured text data as encoded by the large language model, with dimensions consistent with the hidden layer dimensions of the large language model. Entity labels are the names and category identifiers of financial entities identified from the text, including but not limited to stock names, fund names, listed company names, and financial institution names. Sentiment labels are quantitative indicators of the polarity and intensity of the emotion expressed in the text. Polarity is categorized as positive, neutral, and negative, and intensity is represented by a value between 0 and 1, with higher values ​​indicating stronger emotions.

[0071] For example, the computer device concatenates the 768-dimensional semantic enhancement vector obtained in step S21 with the original encoding of a post's text to form a 1536-dimensional composite input representation. Simultaneously, it loads a dynamic set of prompt words containing the new definition of arbitrage in the Beijing Stock Exchange's IPOs. After evolutionary correction, the large language model parses the post and outputs enhanced semantic information. The semantic vector after ontology evolution correction is a 1024-dimensional vector, and the entity labels include two entities: a listed company on the Beijing Stock Exchange and a brokerage firm's business department. The sentiment label is positive with an intensity of 0.78, indicating that the post holds a relatively positive attitude towards the relevant stocks.

[0072] S25. Based on the enhanced semantic information, extract the text sentiment time-series features.

[0073] The computer device extracts sentiment tags from the enhanced semantic information obtained in step S24, and constructs text sentiment time-series features based on the sentiment tags. The computer device first separates the sentiment tag corresponding to each piece of unstructured text data from the enhanced semantic information. The sentiment tag contains two pieces of information: polarity category and intensity value.

[0074] Computer equipment arranges all sentiment tags according to the publication time sequence of unstructured text data, forming a time series with timestamps as the index axis and sentiment intensity values ​​as the value axis. The computer equipment then performs smoothing on this time series, using a moving average algorithm to average the intensity values ​​of adjacent time points over a sliding window. The length of the sliding window is set to 3 to 5 time points to eliminate noise interference from sentiment fluctuations in individual texts. The smoothed time series is the text sentiment time series feature. This feature uses fixed-interval time points as sampling nodes, with each node corresponding to a sentiment intensity value, reflecting the overall tendency and intensity of the emotions of the user group on the financial social platform near that time point.

[0075] Computer devices store the time series features of text sentiment as a vector sequence. Each element in the sequence is a two-dimensional vector. The first component is a timestamp, and the second component is a smoothed sentiment intensity value, which facilitates subsequent alignment and calculation with the time series features of image trends.

[0076] For example, a computer extracts 2,000 sentiment tags from the enhanced semantic information of 2,000 posts collected that day, sorts them by posting time, and obtains the original sentiment sequence from 9:30 AM to 3:00 PM. The computer performs a moving average smoothing with a window length of 3. The smoothed value corresponding to 9:30 AM is the arithmetic mean of the sentiment intensities at 9:30 AM, 9:35 AM, and 9:40 AM. The final generated text sentiment time series feature contains 78 sampling points, each spaced 5 minutes apart. For example, the sentiment intensity corresponding to 10:00 AM is 0.72, indicating that the overall sentiment of the user group in the vicinity of that time is positive and of moderate to high intensity.

[0077] In one possible embodiment, the step of performing spatiotemporal alignment processing between the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs includes: S41. Determine the text sentiment time series features as the reference sequence and the image trend time series features as the sequence to be aligned.

[0078] After completing the feature extraction in steps S2 and S3, the computer device obtains text sentiment time series features and image trend time series features, respectively. The computer device sets the text sentiment time series features as the reference sequence and the image trend time series features as the sequence to be aligned. This setting is based on the fact that the publication time of unstructured text data is usually directly recorded by the servers of financial social platforms, possessing high time accuracy and reliability; therefore, using text sentiment time series features as a time benchmark is more reliable. However, screenshots uploaded by users are often captured by the users themselves, and the capture time may be earlier or later than the post publication time. Furthermore, different users have different capture habits, resulting in the starting point and sampling interval of the image trend time series features on the time axis not being completely consistent with the reference sequence.

[0079] After setup, the computer equipment performs format standardization on both the reference sequence and the sequence to be aligned. It maps the timelines of both sequences to a timestamp sequence based on Coordinated Universal Time (UTC), eliminating time base differences caused by different data sources potentially using local time zones. Simultaneously, the computer equipment normalizes the sampling intervals of the two sequences. If the sampling interval of the reference sequence is 5 minutes and that of the sequence to be aligned is 3 minutes, the computer equipment performs downsampling or interpolation on the sequence to be aligned, unifying the sampling intervals to the same preset interval, set to 5 minutes. The standardized sequences are then comparable, providing a consistent input format for subsequent execution of the dynamic time warping algorithm.

[0080] For example, the text sentiment time series features obtained by the computer device contain 78 sampling points from 9:30 AM to 3:00 PM on June 1, 2024, with each sampling point spaced 5 minutes apart, and the time precision is on the minute level. The image trend time series features contain 73 sampling points from 9:25 AM to 2:55 PM on June 1, 2024, with each sampling point spaced approximately 4 minutes apart. The computer device adjusts the sampling interval of the image trend time series features to 5 minutes using a linear interpolation method and converts all timestamps to UTC time, ultimately obtaining a sequence to be aligned with the reference sequence format, containing the same 78 sampling points but with a starting time offset of approximately 5 minutes.

[0081] S42. Using a dynamic time warping algorithm, calculate the minimum distance path between the reference sequence and the sequence to be aligned.

[0082] S43. Based on the minimum distance path, correct the time offset of the image trend time series features.

[0083] The computer equipment employs the Dynamic Time Warping (VTW) algorithm to perform flexible alignment calculations between the standardized reference sequence and the sequence to be aligned. VTW is an algorithm used to measure the similarity between two time series that may differ in length. This algorithm allows the sequences to be non-linearly stretched and bent along the time axis, thereby finding the optimal matching path between the two sequences.

[0084] The computer device first constructs a distance matrix of size M multiplied by N, where M is the number of sampling points in the reference sequence and N is the number of sampling points in the sequence to be aligned. Each element in the distance matrix represents the Euclidean distance between the i-th sampling point in the reference sequence and the j-th sampling point in the sequence to be aligned, which is obtained by calculating the Euclidean distance between the eigenvectors corresponding to the two sampling points. Then, the computer device performs a dynamic programming operation on the distance matrix, starting from the top left corner (row 1, column 1) and progressively calculating the cumulative minimum distance to the bottom right corner (row M, column N). In each step, the computer device is only allowed to move from the current position to one of three directions: the adjacent cell to the right, the adjacent cell below, or the diagonal cell to the bottom right. The distance value of the current cell is added to the cumulative minimum distance of the previous step, and the smallest cumulative distance among the three directions is taken as the cumulative distance value of the current cell.

[0085] After performing dynamic programming, the computer backtracks from the bottom right corner of the matrix to the top left corner to obtain the path with the minimum cumulative distance; this path is called the minimum distance path. The minimum distance path consists of a series of paired points, each pairing point associating a sample point in the reference sequence with a sample point in the sequence to be aligned. Based on the coordinate differences of each paired point in the minimum distance path, the computer calculates the overall time offset of the sequence to be aligned relative to the reference sequence. The time offset is obtained by taking a weighted average of the timestamp differences of all paired points, with the weights allocated according to the cumulative distance contribution of each paired point in the path. Based on the calculated time offset, the computer uniformly corrects the timestamps of each sample point in the sequence to be aligned. If the offset is positive, it indicates that the entire sequence to be aligned is lagging, and each timestamp needs to be shifted forward by the corresponding time. If the offset is negative, it indicates that the entire sequence to be aligned is ahead, and each timestamp needs to be postponed by the corresponding time.

[0086] For example, the reference sequence has 78 sampling points, and the sequence to be aligned also has 78 sampling points. The computer constructs a 78x78 distance matrix. The minimum distance path obtained after dynamic programming shows that the 10th sampling point of the reference sequence corresponds to the 8th sampling point of the sequence to be aligned, the 20th sampling point of the reference sequence corresponds to the 18th sampling point of the sequence to be aligned, and so on, generally showing a pattern where the sequence to be aligned lags behind by approximately two sampling intervals, or 10 minutes. The computer calculates a weighted average of the timestamp differences of all paired points, resulting in a time offset of positive 10 minutes. Therefore, the timestamps of all 78 sampling points in the sequence to be aligned are shifted forward by 10 minutes, initially aligning the corrected sequence to be aligned with the reference sequence on the time axis.

[0087] S44. Based on the corrected time offset, the screenshot is elastically mapped to its corresponding actual time point to obtain aligned text and image data.

[0088] After correcting for the time offset, the computer device elastically maps the original screenshot data to its corresponding actual time point based on the corrected time offset. This elastic mapping means that the computer device, according to the corrected time offset, re-associates each screenshot from its original capture time to the same time point in the text's sentiment time series features, making the screenshots correspond temporally to the market context described in the text.

[0089] The computer equipment first establishes a corrected time offset lookup table, which records the original capture time of each screenshot, the corrected target time point, and the mapping relationship between the two. If the corrected time point exactly matches the original capture time of a screenshot, the computer equipment directly associates that screenshot with that time point. If the corrected time point lies between the original capture times of two adjacent screenshots, the computer equipment uses a time-weighted interpolation method. Based on the distance ratio between the corrected time point and the original times of the two screenshots, it weights and fuses the image trend features of the two screenshots to generate a virtual intermediate state screenshot feature, which is then used as the image representative for that time point.

[0090] The computer device pairs the features of each original or virtual screenshot with the text sentiment features at its corresponding time point, forming text-image data pairs. All text-image data pairs are arranged in the corrected time point order to form aligned text-image data. The computer device performs a consistency check on the aligned text-image data, checking whether the time point deviation between the text and the screenshot in each text-image data pair is within a preset tolerance range. The tolerance range is set to ±2 minutes. Data pairs exceeding the tolerance are marked as abnormal and discarded.

[0091] For example, a screenshot was originally taken at 10:22 AM on June 1, 2024. After correction in step S42, the target time is 10:30 AM, a deviation of 8 minutes, exceeding the tolerance range of ±2 minutes. The computer finds two screenshots adjacent to 10:30 AM, namely images taken at 10:25 AM and 10:35 AM. Based on the equidistant relationship of 5 minutes between 10:30 AM and each of the two screenshots, the trend features of the two screenshots are weighted and fused with a weight of 0.5 to 0.5 to generate a virtual screenshot feature representing the market state at 10:30 AM, and this feature is paired with the corresponding text sentiment feature at 10:30 AM. Similarly, another screenshot was originally taken at 10:30 AM, and after correction, the target time is also 10:30 AM, with a deviation of 0 minutes, within the tolerance range. The computer directly pairs this original screenshot with the text sentiment feature of 10:30 AM.

[0092] S45. Based on the time mapping relationship in the aligned text-image feature pair, obtain the original text and screenshot after time alignment, as the aligned text-image data.

[0093] The computer device reads the time mapping relationship recorded in the aligned text-image feature pair. This time mapping relationship is stored in memory in the form of a correspondence table between data identifier numbers and corrected time points. Each mapping record contains three pieces of information: the data identifier number of the unstructured text data, the data identifier number of the screenshot, and the corrected time point corresponding to both.

[0094] Based on the corresponding table, the computer device retrieves the original unstructured text data and original screenshot data that match each corrected time point from the temporary cache of the local database. The retrieved original text is the unmodified post text, comment, or discussion thread content collected in step S1, and the retrieved original screenshots are the unmodified user-uploaded image files collected in step S1. The computer device binds each set of retrieved original text and original screenshots according to the corrected time point, forming a text and image data unit that is strictly synchronized in the time dimension.

[0095] The computer equipment transfers all text and image data units from the temporary cache to a dedicated alignment data storage area for persistent storage. Simultaneously, it clears the retrieved original data from the temporary cache to free up storage space. The aligned text and image data serves as the input data source for multimodal depth comparison in step S6, and its time synchronization accuracy directly affects the quality of subsequent comparison label generation. After the transfer is complete, the computer equipment generates a data verification report, recording the total number of text and image data units involved in this alignment process, the number of successfully aligned units, the number of anomalies removed due to tolerance exceeding limits, and the final valid number stored in the database.

[0096] For example, the time mapping table records a record where text ID TXT-20240601-001 and screenshot ID IMG-20240601-001a both correspond to the corrected time point of 10:30. Based on this, the computer device extracts the original post text of TXT-20240601-001 and the original time-sharing screenshot of IMG-20240601-001a from the temporary cache and binds them into a set of aligned text and image data. Similarly, another record in the table, where text ID TXT-20240601-002 and screenshot ID IMG-20240601-001b both correspond to the corrected time point of 10:35, is also extracted and bound by the computer device. Finally, assuming a total of 5,000 sets of mapping records are processed on that day, of which 4,850 sets are successfully aligned within the tolerance range and 150 sets are rejected due to excessive deviation, the computer equipment transfers the 4,850 sets of aligned text and image data to the aligned data storage area.

[0097] In one possible embodiment, performing multimodal deep comparison on the aligned text-image data to generate structured comparison labels includes: Step S61: Input the aligned text and image data into the multimodal large model.

[0098] The computer device reads the aligned text and image data generated in steps S4 and S5 from the aligned data storage area. The aligned text and image data is stored in paired groups, with each group containing one original text and one original screenshot, which are bound together by a unified timestamp. The computer device performs preprocessing operations on each group of aligned text and image data, including denoising the original text and normalizing the size of the original screenshot.

[0099] Noise reduction refers to the process by which computer equipment filters and replaces special symbols, repeated characters, and meaningless internet slang in the original text, retaining the text content that can express clear semantics. Size normalization refers to the process by which computer equipment adjusts the pixel size of the original screenshot to the standard input size required by the multimodal large model. The standard input size is set to 224 pixels wide by 224 pixels high. During the adjustment process, the aspect ratio of the screenshot is kept unchanged, and excess areas are cropped proportionally.

[0100] After preprocessing, the computer device encodes the text and image portions of each set of aligned text-image data into an input format acceptable to the multimodal large model. The text portion is converted into a text token sequence by a tokenizer, and the image portion is converted into an image feature vector by a visual encoder. The computer device concatenates and combines the text token sequence and image feature vector according to the input interface specification of the multimodal large model to form a multimodal joint input representation. The computer device sends this multimodal joint input representation to the inference interface of the multimodal large model one by one. After receiving the input, the multimodal large model performs cross-modal understanding and inference on each set of aligned text-image data in sequence.

[0101] The multimodal large model is a pre-trained deep neural network model with dual capabilities of text understanding and image understanding. During the training phase, this model was jointly trained using large-scale text-image pairing data, enabling it to extract semantic intent from text and identify visual elements from images. When the computer device performs this step, the inference mode of the multimodal large model is set to contrastive analysis mode. In this mode, the output of the multimodal large model focuses on determining the degree of semantic correlation between the input text and the input image.

[0102] For example, a computer device reads a set of aligned text and image data from an aligned data storage area. This set of data contains the original text "I made a lot of money buying [stock name] today, highly recommended," and a screenshot showing the intraday price movement of that stock. The computer device denoises the text to obtain the "I made a lot of money buying [stock name] today, highly recommended," and normalizes the screenshot to obtain a standard 224-pixel image. The computer device encodes the text into a 128-bit text token sequence and the image into a 768-dimensional image feature vector. These two are concatenated to form a multimodal joint input representation, which is then fed into a multimodal large model. The computer device then feeds all 320 sets of aligned text and image data for the day into the multimodal large model one by one in the same manner to perform inference.

[0103] Step S62: Using a multimodal large model, extract investors' subjective opinions from the text data in the aligned text and image data, and extract objective profit and loss facts from the image data.

[0104] After the computer device completes inference in the multimodal large model, it extracts textual and image-based information from the output. The computer device extracts investor subjective statements from the textual output. Investor subjective statements refer to the semantic description obtained by the multimodal large model after structured summarization of investor sentiment, investment intentions, and subjective judgments about the future trend of the underlying assets reflected in the textual data.

[0105] Specifically, the computer device extracts three core elements from the text output of the multimodal large model: sentiment polarity, behavioral direction, and expected judgment. Sentiment polarity represents the investor's current attitude towards the underlying asset, categorized into positive, neutral, and negative sentiment. Behavioral direction represents the investor's operational intentions revealed in the text, categorized into buying, holding, and selling tendencies. Expected judgment represents the investor's subjective prediction of the underlying asset's future price movement, categorized into bullish, neutral, and bearish expectations. The computer device combines these three core elements into a complete record of the investor's subjective statement.

[0106] Simultaneously, the computer equipment extracts objective profit and loss facts from the image-side output of the multimodal large model. Objective profit and loss facts refer to the objective description obtained by the multimodal large model after visual recognition and numerical reading of the financial data charts displayed in the screenshot. The multimodal large model identifies candlestick charts, intraday charts, or position profit and loss screenshots in the screenshots, extracting key numerical information such as the current price change relative to the purchase price, the specific amount or percentage of position profit and loss, and the time interval of the chart.

[0107] The computer equipment extracts two core elements from the image output: the actual price fluctuation and the profit / loss status. The actual price fluctuation is the percentage change in the underlying asset price shown in the screenshot relative to a reference time point, expressed numerically: positive values ​​indicate an increase, and negative values ​​indicate a decrease. The profit / loss status is determined based on the actual price fluctuation and a preset profit / loss threshold: a fluctuation greater than or equal to +2% is considered profitable, a fluctuation less than or equal to -2% is considered a loss, and a fluctuation between -2% and +2% is considered break-even. The computer equipment combines the actual price fluctuation and the profit / loss status into a complete and objective record of the profit / loss situation.

[0108] For example, after analyzing the text "I bought a certain stock today and made a lot of money, I strongly recommend everyone to follow suit," the multimodal large model extracts the following subjective investor sentiment: positive emotional polarity, buying tendency, and bullish expectation. Simultaneously, after visually recognizing the corresponding screenshot, the multimodal large model extracts the objective profit / loss facts: an actual price change of +5.3%, indicating a profitable state. As another example, in another set of aligned text and image data, the text is "This stock is completely ruined, run!" The multimodal large model extracts the following subjective investor sentiment: negative emotional polarity, selling tendency, and bearish expectation. The corresponding screenshot extracts the following objective profit / loss facts: an actual price change of -3.8%, indicating a loss.

[0109] Step S63: Based on the investor's subjective statement and the objective profit and loss facts, generate structured comparison labels. The structured comparison labels are used to characterize the consistency relationship between the investor's subjective statement and the objective profit and loss facts.

[0110] In step S62, after obtaining the investor's subjective statement and objective profit and loss facts, the computer device cross-compares the two to generate a structured comparison label. The structured comparison label is a structured data record containing multiple fields, used to quantitatively characterize the degree of consistency between the investor's subjective statement and the objective profit and loss facts.

[0111] The computer system first matches the investor's subjective statements—emotional polarity, behavioral direction, and expected judgment—with the objective profit / loss facts—the actual price fluctuations and profit / loss status—one by one. The computer system sets the following consistency judgment rules: When the emotional polarity is positive and the profit / loss status is profitable, or when the emotional polarity is negative and the profit / loss status is loss-making, the computer system determines that the two are consistent in terms of both emotion and fact, and records it as consistent. When the emotional polarity is positive and the profit / loss status is loss-making, or when the emotional polarity is negative and the profit / loss status is profitable, the computer system determines that the two are contradictory in terms of both emotion and fact, and records it as contradictory. When the emotional polarity is neutral, regardless of the profit / loss status, the computer system records it as pending.

[0112] The computer system further combines the behavioral direction with the expected judgment and the actual price fluctuation. When the behavioral direction is a buying tendency, the expected judgment is bullish, and the actual price fluctuation is positive, the computer system determines that the three are consistent and records it as highly consistent. When the behavioral direction is a selling tendency, the expected judgment is bearish, and the actual price fluctuation is negative, it is also recorded as highly consistent. When there is an inverse relationship between the combination of behavioral direction and expected judgment and the actual price fluctuation, the computer system records it as a significant deviation.

[0113] The computer equipment calculates a consistency score based on the above judgment results. The consistency score is calculated as follows: the consistency score equals the number of consistent items divided by the total number of judgment items, multiplied by 100. The total number of judgment items is 5, namely, 1 item in the emotion and fact dimension, 1 item in the behavior and price fluctuation dimension, 1 item in the expectation and price fluctuation dimension, 1 item in the emotion and behavior dimension, and 1 item in the expectation and behavior dimension. The consistency score ranges from 0 to 100, with a higher score indicating a stronger consistency between the investor's subjective statement and the objective profit and loss facts.

[0114] The computer device integrates the consistency score, various judgment results, and the original content of investors' subjective statements and objective profit and loss facts into a structured comparison label. The structured comparison label includes six fields: text sentiment, image fact, sentiment fact consistency, behavioral rise / fall consistency, expected rise / fall consistency, and overall consistency score. The computer device aggregates and stores the structured comparison labels corresponding to all aligned text and image data, which serve as input data for subsequent step S7, the group behavior analysis.

[0115] For example, regarding the first set of aligned text and image data mentioned above, investors' subjective sentiments were positive, indicating a buying tendency and bullish expectations, while the objective profit / loss facts were a positive 5.3% and a profitable state. After item-by-item evaluation, the computer equipment found that sentiment and facts were consistent, behavior and price fluctuations were consistent, expectations and price fluctuations were consistent, sentiment and behavior were consistent, and expectations and behavior were consistent—all five items were consistent, resulting in a consistency score of 5 divided by 5 multiplied by 100, which equals 100. The overall consistency score field in the structured comparison labels generated by the computer equipment was 100, and all consistency fields were marked as consistent. Regarding another set of data, the text stated "This stock is completely ruined, run!" while the screenshot showed a slight increase of 1.2%. The computer equipment determined that the negative sentiment polarity contradicted the profitable state, and the selling behavior contradicted the positive actual price fluctuation. The consistency score was 1 divided by 5 multiplied by 100, which equals 20. The overall consistency score field in the structured comparison labels was 20, the sentiment-fact consistency field was marked as contradictory, and the behavior-price fluctuation consistency field was marked as contradictory.

[0116] In one possible embodiment, constructing a training set based on the aligned text-image feature pairs and the structured contrast labels includes: S71. The text sentiment time series features and image trend time series features in the aligned text-image feature pair are fused to form multimodal input features.

[0117] The computer device retrieves all aligned text-image feature pairs generated in steps S4 and S5 from the aligned data storage area. Each text-image feature pair contains one text sentiment time-series feature and one image trend time-series feature, which have been strictly synchronized in the time dimension through the spatiotemporal alignment process in step S4. The computer device performs a multimodal feature fusion operation on each text-image feature pair, merging the text sentiment time-series feature and the image trend time-series feature into a unified multimodal input feature vector.

[0118] The computer device employs a feature-level fusion strategy to perform the fusion operation. Feature-level fusion refers to directly concatenating or weighting the feature vectors of two modalities at the feature representation level to form a joint feature vector with higher dimensionality but more complete information. The computer device first performs dimension alignment processing on the text sentiment time series features and the image trend time series features respectively. After processing in step S3, the text sentiment time series features are a two-dimensional matrix with time steps as rows and sentiment feature dimensions as columns, assuming its dimension is T×d. t Where T is the number of time steps, d t The text sentiment feature dimension is set to 128 dimensions. After processing in step S3, the image trend time series features are a two-dimensional matrix with time steps as rows and trend feature dimensions as columns, with dimensions T×d. i , where d i The dimension of image trend features is set to 128.

[0119] The computer device performs global average pooling on both matrices along the time dimension, averaging the feature vectors at each time step along the time axis to obtain a fixed-length one-dimensional feature vector. The text sentiment time series features, after global average pooling, yield a 128-dimensional text global feature vector, and the image trend time series features, after global average pooling, yield a 128-dimensional image global feature vector.

[0120] Subsequently, the computer device concatenates the text global feature vector with the image global feature vector in a first-to-last concatenation manner, that is, first arranging the 128 elements of the text global feature vector, and then arranging the 128 elements of the image global feature vector to form a 256-dimensional concatenated feature vector.

[0121] Building upon the concatenation process, the computer device further introduces a cross-modal attention mechanism to weight and optimize the concatenated feature vector. This mechanism dynamically weights each dimension of the concatenated feature vector by calculating the correlation weights between text and image features, thus assigning higher weights to feature dimensions that contribute more to consistency determination between the two modalities. The computer device sets up a learnable attention weight matrix with dimensions 256×256, initialized using a random normal distribution. The device then performs matrix multiplication between the concatenated feature vector and the attention weight matrix, resulting in a 256-dimensional attention-weighted feature vector. This attention-weighted feature vector is then used as the final multimodal input feature.

[0122] The computer device performs the above fusion operation on each of the text and image feature pairs to obtain a multimodal input feature set with the same number of text and image feature pairs.

[0123] For example, a computer device retrieves a set of aligned text-image feature pairs. The text sentiment time series features are a 78-time-step × 128-dimensional matrix, and the image trend time series features are also a 78-time-step × 128-dimensional matrix. The computer device performs global average pooling on the text matrix, averaging the 78-row, 128-column matrix along the row directions to obtain a 128-dimensional vector. Assume its first 5 elements are [0.72, 0.65, 0.58, 0.81, 0.49]. The same operation is performed on the image matrix to obtain a 128-dimensional vector, assuming its first 5 elements are [0.68, 0.71, 0.55, 0.79, 0.52]. The computer device concatenates the two to obtain a 256-dimensional vector, with the first 128 dimensions representing the text global features and the last 128 dimensions representing the image global features. After cross-modal attention weighting, assuming the first 5 elements become [0.74, 0.67, 0.60, 0.83, 0.51], this 256-dimensional vector is the multimodal input feature of this set of text-image feature pairs.

[0124] S72. Convert the structured comparison label into a classification label, and use the classification label as the supervision label for the training sample.

[0125] The computer device retrieves all structured comparison labels from the output storage area of ​​step S6. Each structured comparison label contains six fields: text sentiment, image fact, sentiment-fact consistency, behavioral rise-fall consistency, expected rise-fall consistency, and overall consistency score. The computer device performs a label transformation operation on each structured comparison label, converting it from a multi-field descriptive label into a single classification label.

[0126] A classification labeling system is established for computer equipment. This system comprises four categories: highly consistent, basically consistent, partially contradictory, and significantly contradictory. The classification labels are determined based on the overall consistency score field in the structured comparison label. The score ranges for the four categories are as follows: When the overall consistency score is greater than or equal to 80, it is converted to a highly consistent label. When the overall consistency score is greater than or equal to 60 and less than 80, it is converted to a basically consistent label. When the overall consistency score is greater than or equal to 40 and less than 60, it is converted to a partially contradictory label. When the overall consistency score is less than 40, it is converted to a significantly contradictory label.

[0127] If any consistency field in the structured comparison labels contains pending markers, the computer device does not include it in the calculation of the overall consistency score. Instead, it recalculates the revised consistency score based on the number of valid judgments. The revised consistency score is equal to the number of consistent items divided by the number of valid judgments, multiplied by 100. The computer device performs the aforementioned classification transformation based on the revised consistency score.

[0128] In addition, the computer device performs one-hot encoding on the classification labels, converting the category names into machine-readable numerical vectors. The one-hot encoding methods are as follows: highly consistent encoding is [1, 0, 0, 0], basically consistent encoding is [0, 1, 0, 0], partially contradictory encoding is [0, 0, 1, 0], and significantly contradictory encoding is [0, 0, 0, 1]. The computer device uses the one-hot encoded vectors as the supervision labels for the training samples.

[0129] The computer equipment performs the above transformation and encoding operations on all structured contrast labels one by one to obtain a set of supervision labels that is the same number as the multimodal input features.

[0130] For example, a computer retrieves a structured comparison label with an overall consistency score of 100, where all consistency fields are marked as consistent. The computer determines that 100 is greater than or equal to 80, converting it to the highly consistent category with a one-hot encoding of [1, 0, 0, 0]. Another example: a structured comparison label has an overall consistency score of 20, with the emotional fact consistency field marked as contradictory, the behavioral rise / fall consistency field marked as contradictory, and the expected rise / fall consistency field marked as pending. After excluding the pending items, the computer has 3 valid judgment items and 1 consistent item, resulting in a corrected consistency score of 1 divided by 3 multiplied by 100, approximately 33.3. The computer determines that 33.3 is less than 40, converting it to the significantly contradictory category with a one-hot encoding of [0, 0, 0, 1]. Yet another example: a structured comparison label has an overall consistency score of 70, converting it to basically consistent with a one-hot encoding of [0, 1, 0, 0].

[0131] S73. The multimodal input features and the supervision labels are used to form training samples, which are then aggregated to form the training set.

[0132] After completing steps S71 and S72, the computer device pairs and combines each group of multimodal input features with its corresponding supervision label to form a complete training sample. Each training sample consists of a 256-dimensional multimodal input feature vector and a 4-dimensional one-hot encoded supervision label vector, and the two are bound together by the same sample number.

[0133] The computer equipment performs data quality checks on all training samples. The data quality checks include the following three parts: First, an integrity check: the computer equipment iterates through all training samples, confirming that each sample's multimodal input feature vector does not contain null values ​​or infinity, and that the supervision label vector has exactly one element of 1 and the other three elements of 0. If a sample does not meet these conditions, the computer equipment marks it as invalid and removes it from the training set. Second, a class imbalance check: the computer equipment counts the number of samples in each of the four categories and calculates the proportion of each category to the total number of samples. If the proportion of a certain category is lower than 10% or higher than 50%, the computer equipment determines that the training set has a class imbalance problem and records the imbalanced category's identifier. Third, a consistency check: the computer equipment randomly selects 5% of the training samples and manually reviews the correspondence between their multimodal input features and supervision labels to ensure correctness. If the review finds an error rate exceeding 1%, the computer equipment re-performs pairwise verification on all samples.

[0134] For training sets with class imbalance, the computer uses an oversampling strategy to balance the classes. Oversampling involves randomly copying classes with fewer samples to increase their sample size to a level similar to the class with the most samples. The computer sets an expansion target of no more than 20% difference in sample size between classes.

[0135] After data quality checks and balancing are completed, the computer system sorts all valid training samples according to their sample numbers, forming an ordered training sample list. This training sample list is then divided into a training subset and a validation subset. The training subset comprises 80% of all samples, and the validation subset comprises 20%. The partitioning method uses stratified randomization, meaning that samples are randomly selected from each category at an 8:2 ratio to ensure that the proportions of each category in the training and validation subsets remain consistent with the original training set.

[0136] The computer device stores the training subset as the main training file and the validation subset as the validation file, each labeled with a different file identifier. The main training file contains the multimodal input feature matrix and supervision label matrix of all training subset samples. The matrix format is a numerical array, stored in binary format to improve reading efficiency. The computer device records the total number of training samples, the number of samples in each class, feature dimensions, number of label classes, generation time, and data quality check results in the metadata of the main training file.

[0137] For example, the computer device obtained 5000 aligned text-image feature pairs and their corresponding structured contrast labels. After fusion in step S71, 5000 256-dimensional multimodal input feature vectors were obtained. After transformation in step S72, 5000 4-dimensional one-hot encoded supervision label vectors were obtained. Data quality checks revealed that 3 samples with null values ​​were removed, leaving 4997 valid samples. Balance checks found that 2400 samples (48.0%) were highly consistent, 1300 samples (26.0%) were basically consistent, 800 samples (16.0%) were partially contradictory, and 497 samples (9.9%) were significantly contradictory. The proportion of significantly contradictory categories was less than 10%. The computer device oversampled the significantly contradictory categories, randomly copying samples of this category to 960 samples, so that the proportion of each category was between 19% and 25%. From the final 4997 plus 63 equals 5060 samples, the computer equipment randomly divided the samples stratified, selecting 80% (4048 samples) as the training set and 20% (1012 samples) as the validation set. The main file of the training set records metadata information such as the total number of samples (4048), feature dimensions (256), label categories (4), generation time, and whether the quality check was passed.

[0138] In another embodiment, the financial public opinion analysis model is a classifier based on a multimodal neural network. This classifier uses the training set constructed in step S7 as training data, the multimodal input features generated in step S71 as input, and the classification labels generated in step S72 as supervision signals, and is trained through end-to-end supervised learning. The trained classifier is used to perform real-time public opinion consistency determination on newly collected text and image data, and outputs the public opinion consistency classification result. This embodiment is executed by a computer device, which includes a processor, memory, and a graphics processor. The graphics processor is used to accelerate the training and inference operations of the multimodal neural network.

[0139] I. Overall architecture of the classifier.

[0140] The computer-based classifier using a multimodal neural network employs a dual-tower fusion architecture, consisting of four core modules: a text feature extraction tower, an image feature extraction tower, a cross-modal fusion layer, and a classification decision layer. The text feature extraction tower performs deep semantic encoding on the input text sentiment time-series features, the image feature extraction tower performs deep visual encoding on the input image trend time-series features, the cross-modal fusion layer integrates the feature representations output from the two towers, and the classification decision layer outputs the final sentiment consistency classification result based on the fused features.

[0141] The text feature extraction tower employs a deep neural network structure based on a Transformer encoder. This structure comprises six Transformer encoder layers, each containing two sub-modules: a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism has eight attention heads, each with a dimension of 64, resulting in a total dimension of 512 for each encoder layer's self-attention module. The feedforward neural network consists of two fully connected layers: the first maps the 512-dimensional vector to 2048-dimensional vectors, and the second maps the 2048-dimensional vectors back to 512-dimensional vectors, with the GELU activation function used between the two layers. The input dimension of the text feature extraction tower is 128, which is the dimension of the global text feature vector after global average pooling in step S71. A linear projection layer is added at the input end of the text feature extraction tower to map the 128-dimensional input to 512-dimensional vectors, serving as the input embedding for the Transformer encoder.

[0142] The image feature extraction tower employs a deep neural network structure based on the Visual Transformer, which contains 12 Visual Transformer encoder layers. Each encoder layer also includes a multi-head self-attention mechanism and a feedforward neural network. The multi-head self-attention mechanism has 12 attention heads, each with a dimension of 64, resulting in a total dimension of 768 for the self-attention modules in each encoder layer. The feedforward neural network consists of two fully connected layers: the first layer maps the 768-dimensional array to 3072-dimensional arrays, and the second layer maps the 3072-dimensional arrays back to 768-dimensional arrays, with the GELU activation function used between the two layers. The input dimension of the image feature extraction tower is 128, which is the dimension of the global feature vector of the image after global average pooling in step S71. The image feature extraction tower incorporates a linear projection layer at the input, mapping the 128-dimensional input to 768-dimensional arrays, which serves as the input embedding for the Visual Transformer encoder.

[0143] The cross-modal fusion layer employs a cross-attention mechanism. This mechanism uses the output of the text feature extraction tower as the query vector and the output of the image feature extraction tower as the key and value vectors. By calculating the attention weights between the query vector and the key vector, and then weighted summing the value vectors, the text-guided image enhancement features are obtained. The output dimension of the cross-attention mechanism is set to 512. The computer device performs residual concatenation and layer normalization on the original output of the text feature extraction tower and the output of the cross-attention mechanism to obtain a 512-dimensional fused feature vector.

[0144] The classification decision layer consists of three fully connected layers and a softmax output layer. The first fully connected layer maps the 512-dimensional fused feature vector to 256 dimensions, using ReLU activation and Dropout regularization with a Dropout ratio of 0.3. The second fully connected layer maps the 256 dimensions to 128 dimensions, also using ReLU activation and Dropout regularization with a Dropout ratio of 0.3. The third fully connected layer maps the 128 dimensions to 4 dimensions, corresponding to the four classification categories. The softmax output layer normalizes the 4-dimensional output, converting it into a probability distribution for the four categories, with the sum of the probabilities equal to 1.

[0145] II. Input layer of the classifier.

[0146] The input layer of the classifier receives the multimodal input features generated in step S71. The multimodal input features are 256-dimensional attention-weighted feature vectors. The first 128 dimensions of this vector correspond to the attention-weighted representation of the global text features, and the last 128 dimensions correspond to the attention-weighted representation of the global image features.

[0147] Before feeding multimodal input features into the classifier, the computer performs input preprocessing. Input preprocessing includes two steps: feature standardization and feature clipping. Feature standardization involves applying Z-score standardization to each dimension of the 256-dimensional multimodal input feature vector. This involves subtracting the mean of that dimension from the mean of all training samples and then dividing by the standard deviation of that dimension, making the distribution of each dimension approximately a standard normal distribution with a mean of 0 and a standard deviation of 1. Feature clipping involves truncating the standardized feature vector, limiting the value of each dimension to the range of -3 to +3. Values ​​outside this range are truncated to either -3 or +3, respectively, to eliminate the interference of extreme outliers on model training.

[0148] The input layer receives a preprocessed 256-dimensional standardized feature vector and splits it into two 128-dimensional sub-vectors according to a preset segmentation rule. The first 128-dimensional sub-vector is fed into the linear projection layer of the text feature extraction tower, and the second 128-dimensional sub-vector is fed into the linear projection layer of the image feature extraction tower. The input layer records the segmentation location information during the splitting process to ensure that the same splitting operation is performed on new data in subsequent inference stages.

[0149] The computer device sets the batch size of the input layer to 64, meaning that 64 multimodal input feature vectors are grouped into a batch and fed into the classifier simultaneously for parallel computation. The computer device sets the input layer's sequence padding strategy to dynamic padding, meaning that for batches with fewer than 64 samples, the computer device does not perform zero-padding, but instead calculates based on the actual number of samples in the batch, avoiding the introduction of invalid padding noise.

[0150] III. The training process of the classifier.

[0151] The computer device performs supervised training on the classifier using the training set master file constructed in step S7. The training process employs a mini-batch stochastic gradient descent optimization algorithm, with 100 epochs per training round. The initial learning rate is set to 0.001, and a cosine annealing learning rate scheduling strategy is used, meaning the learning rate gradually decays from 0.001 to 0.00001 according to a cosine function as the training rounds increase. The computer device sets a weight decay coefficient of 0.01 for L2 regularization to prevent overfitting. The computer device sets a gradient clipping threshold of 1.0; when the gradient norm exceeds 1.0, the gradient is truncated to prevent gradient explosion.

[0152] At the beginning of each epoch, the computer randomly shuffles the training subset to ensure that the input order of samples is different in each epoch. The computer then sequentially draws samples from the training subset in batches of 64, feeding the multimodal input features of each batch into the input layer of the classifier, and obtaining a 4-dimensional probability output through forward propagation. The computer then calculates the cross-entropy loss using this 4-dimensional probability output and the corresponding one-hot encoded supervision label. The cross-entropy loss is calculated as the sum of the dot product of the negative supervision label vector and the logarithm of the probability output vector.

[0153] After calculating the loss for the current batch, the computer performs backpropagation to calculate the gradient of the loss function with respect to all classifier parameters. The computer then updates all classifier parameters based on the gradient and the learning rate. At the end of each epoch, the computer evaluates the classification performance of the current model using a validation subset. Evaluation metrics include classification accuracy, precision, recall, and F1 score. The computer records the optimal model parameters on the validation subset as the final training result.

[0154] The computer system is equipped with an early stopping mechanism: when the classification accuracy on the validation subset no longer improves for 10 consecutive epochs, the computer system terminates training and retains the optimal model parameters. The computer system has a minimum learning rate of 0.00001, and once the learning rate decays to this value, it will not decrease further.

[0155] IV. Output layer of the classifier.

[0156] The classifier's output layer is a Softmax output layer. This layer receives a 4-dimensional real vector from the output of the third fully connected layer of the classification decision layer and transforms it into a probability distribution for four categories. The four categories correspond to highly consistent, basically consistent, partially contradictory, and significantly contradictory, respectively, corresponding to the 1st, 2nd, 3rd, and 4th dimensions of the output vector.

[0157] After obtaining the 4-dimensional probability vector from the Softmax output layer, the computer device performs a classification operation. The classification operation adopts the maximum probability principle, meaning the computer device compares the four probability values ​​and selects the category corresponding to the dimension with the highest probability value as the final public opinion consistency classification result. If multiple dimensions have the same probability value, and all are the maximum value, the computer device selects the category with the smallest dimension number as the classification result.

[0158] In addition, the computer device also outputs confidence level information. Confidence level is defined as the maximum probability value itself, ranging from 0 to 1. When the confidence level is greater than or equal to 0.8, the computer device marks the classification result as a high-confidence decision. When the confidence level is between 0.6 and 0.8, it is marked as a medium-confidence decision. When the confidence level is less than 0.6, it is marked as a low-confidence decision. The computer device outputs the classification result category, corresponding probability value, and confidence level together.

[0159] The computer device also outputs detailed probability distributions for each category for subsequent analysis. Each detailed probability distribution contains four probability values, corresponding to the posterior probabilities of the four categories. The computer device packages the classification result category, the four probability values, the confidence level, and the sample ID of the input sample into a structured output record and stores it in the output result storage area.

[0160] In one embodiment, the computer device performs supervised training on a pre-defined financial public opinion analysis model. Through repeated forward propagation calculations and backpropagation parameter updates, the model gradually learns the mapping relationship between multimodal input features and public opinion consistency classification results, and finally obtains a trained financial public opinion analysis model with accurate judgment capabilities.

[0161] Step S81: Input the multimodal input features into the financial public opinion analysis model to obtain predicted labels.

[0162] The computer device reads all multimodal input features of the training subset from the main training set file constructed in step S7. The training subset contains 4048 standardized 256-dimensional multimodal input feature vectors, each of which has undergone quality checking and preprocessing in step S73. The computer device then feeds the multimodal input features from the training subset into the input layer of the financial sentiment analysis model in batches.

[0163] The computer equipment is configured with a training batch size of 64, meaning that 64 multimodal input feature vectors are randomly selected from the training subset each time to form a training batch. The batch selection method is set to random sampling without replacement, meaning each sample is selected only once per epoch, ensuring that all training samples are used within an epoch. Since the training subset contains 4048 samples, the computer equipment will generate 4048 divided by 64, which equals 63 complete batches per epoch.

[0164] The computer equipment combines the 64 256-dimensional multimodal input feature vectors from each batch into a 64×256 input matrix, which is then fed into the input layer of the financial public opinion analysis model. Following the segmentation rules defined in step S7, the input layer splits each 256-dimensional vector into a first 128-dimensional text sub-vector and a second 128-dimensional image sub-vector. The text sub-vectors are then fed into the linear projection layer of the text feature extraction tower and mapped to a 512-dimensional embedding vector, while the image sub-vectors are fed into the linear projection layer of the image feature extraction tower and mapped to a 768-dimensional embedding vector.

[0165] The text feature extraction tower performs a 6-layer Transformer encoder layer-by-layer encoding on the 512-dimensional text embedding vector. Each encoder layer first calculates the dependencies between time steps within the text embedding vector using an 8-head self-attention mechanism, obtaining a 512-dimensional self-attention output. Then, a feedforward neural network maps the 512-dimensional vector to 2048-dimensional and back to 512-dimensional, followed by GELU activation and residual connections to output a 512-dimensional deep text feature vector. After the 6-layer encoding is complete, the text feature extraction tower outputs a 64×512 text feature matrix, meaning each of the 64 samples corresponds to a 512-dimensional deep text feature vector.

[0166] The image feature extraction tower performs layer-by-layer encoding on the 768-dimensional image embedding vector using a 12-layer visual Transformer encoder. Each encoder layer calculates the dependencies between time steps within the image embedding vector using a 12-head self-attention mechanism, resulting in a 768-dimensional self-attention output. Subsequently, a feedforward neural network maps the 768-dimensional vector to 3072-dimensional and then back to 768-dimensional. After GELU activation and residual connections, a 768-dimensional deep image feature vector is output. After the 12-layer encoding, the image feature extraction tower outputs a 64×768 image feature matrix, meaning each of the 64 samples corresponds to a 768-dimensional deep image feature vector.

[0167] The cross-modal fusion layer uses the deep text feature matrix as the query vector and the deep image feature matrix as the key and value vectors, performing cross-attention calculation. The cross-attention calculation method is as follows: First, the query vector and key vector are multiplied by a matrix to obtain a 64×64 attention weight matrix. This matrix is ​​then Softmax normalized to obtain normalized attention weights. Next, the normalized attention weights are multiplied by the value vector to obtain a 64×768 weighted feature matrix. The computer device maps this weighted feature matrix through a linear projection layer to a 64×512 fusion feature matrix, which is then residually connected to the deep text feature matrix and subjected to layer normalization to obtain the final 64×512 fusion feature matrix.

[0168] The classification decision layer performs a three-layer fully connected mapping on the 64×512 fused feature matrix. The first layer maps 512 dimensions to 256 dimensions, the second layer maps 256 dimensions to 128 dimensions, and the third layer maps 128 dimensions to 4 dimensions. After each fully connected layer, ReLU activation and Dropout regularization are performed sequentially, with the Dropout ratio set to 0.3. The third fully connected layer outputs a 64×4 real matrix, meaning that each of the 64 samples corresponds to a 4-dimensional unnormalized score vector.

[0169] The computer performs a Softmax normalization operation on the 64×4 real number matrix, converting the four real numbers in each row into four probability values, such that the sum of the four probability values ​​in each row is 1. The normalized result is a 64×4 probability matrix, which is the predicted label matrix for the current batch. Each row in the predicted label matrix corresponds to one of the four probability distributions for a sample: highly consistent, basically consistent, partially contradictory, and significantly contradictory.

[0170] The computer equipment performed the above forward propagation process on all 63 batches one by one to obtain a complete prediction label matrix covering all 4048 training samples.

[0171] Step S82: Calculate the loss function value between the predicted label and the corresponding supervised label in the training set.

[0172] The computer device reads 64 supervision labels corresponding to the current batch from the main training set file. Each supervision label is a 4-dimensional one-hot encoded vector, for example, highly consistent corresponds to [1, 0, 0, 0], basically consistent corresponds to [0, 1, 0, 0], partially contradictory corresponds to [0, 0, 1, 0], and significantly contradictory corresponds to [0, 0, 0, 1]. The computer device arranges the 64 supervision labels into a 64×4 supervision label matrix, which corresponds one-to-one with the 64×4 prediction label matrix obtained in step S81 in the sample dimension.

[0173] The computer equipment uses the cross-entropy loss function to calculate the loss value of the current batch. The formula for calculating the cross-entropy loss function is: for each sample, multiply each element of its supervised label vector by the logarithm of the corresponding element of its predicted label vector, take the negative value, sum the results, and then calculate the average over all samples in the batch. Specifically, for the i-th sample, the supervised label vector y with a negative loss value... i The dot product with the logarithm of the predicted probability vector pi, i.e., L i equal- , where j ranges from 1 to 4. The batch loss value L is equal to the arithmetic mean of the loss values ​​of 64 samples.

[0174] To prevent the prediction probability from having zero values ​​that would lead to an undefined logarithm, the computer performs a numerical clamping operation on the prediction probability vector before calculating the logarithm, limiting each probability value to e.-8 Within the range of 1 to 1. When a certain probability value is less than e -8 It will be raised to e -8 When it is greater than 1, reduce it to 1.

[0175] The computer device introduces a label smoothing regularization term on top of the cross-entropy loss. Label smoothing refers to replacing 1s in the one-hot encoded supervised label vector with 1 minus the smoothing coefficient ε, and replacing 0s with ε divided by the number of classes minus 1. The smoothing coefficient ε is set to 0.1, and the number of classes is 4. Taking highly consistent classes as an example, the original label [1, 0, 0, 0] becomes [0.925, 0.025, 0.025, 0.025] after label smoothing. The purpose of label smoothing is to prevent the model from being overconfident in the training data and improve the model's generalization ability. The computer device uses the smoothed labels to calculate the cross-entropy loss.

[0176] In addition, the computer device calculates the L2 regularization loss term. The L2 regularization loss term is equal to the sum of the squares of all trainable parameters of the model multiplied by half of the weight decay factor. The weight decay factor is set to 0.01. The computer device adds the cross-entropy loss to the L2 regularization loss to obtain the total loss function value for the current batch.

[0177] The computer performs loss calculations on each of the 63 batches one by one, obtaining the loss function value for each batch. The computer records the average of the loss values ​​of all batches within each epoch as the training loss for that epoch.

[0178] Step S83: With the goal of minimizing the loss function value, iteratively optimize the model parameters of the financial public opinion analysis model.

[0179] The computer device optimizes the financial sentiment analysis model by minimizing the total loss function calculated in step S82, iteratively updating all trainable parameters. The model's trainable parameters include all weight matrices and bias vectors of the 6-layer Transformer encoder in the text feature extraction tower, all weight matrices and bias vectors of the 12-layer visual Transformer encoder in the image feature extraction tower, the query projection matrix, key projection matrix, value projection matrix, and weights and biases of the linear projection layer in the cross-modal fusion layer's cross-attention mechanism, and the weight matrices and bias vectors of the three fully connected layers in the classification decision layer. The total number of trainable parameters is approximately 28 million.

[0180] The computer system uses the AdamW optimizer to perform parameter updates. The AdamW optimizer is an improved version of the Adam optimizer that decouples from weight decay, enabling independent L2 weight decay while adaptively adjusting the learning rate. The initial learning rate of the AdamW optimizer is set to 0.001, the decay coefficient β1 for the first moment estimation is set to 0.9, the decay coefficient β2 for the second moment estimation is set to 0.999, the weight decay coefficient is set to 0.01, and the ε parameter is set to 1e-8 to prevent division by zero errors.

[0181] The computer equipment employs a cosine annealing learning rate scheduling strategy. Cosine annealing refers to the learning rate gradually decreasing from an initial value of 0.001 to a minimum value of 0.00001 according to a cosine function as the training epoch increases. Specifically, the learning rate η in the t-th epoch... t This is equal to the minimum learning rate plus half of that, multiplied by the difference between the initial and minimum learning rates, and then multiplied by (1 plus cos(πt / T)), where T is the total number of training epochs set to 100. This strategy allows the model to converge quickly with a larger learning rate in the early stages of training, and then fine-tuned with a smaller learning rate in the later stages.

[0182] The computer device sets the gradient clipping threshold to 1.0. After calculating the gradient in each backpropagation, the computer device calculates the L2 norm of all parameter gradients. If the gradient norm is greater than 1.0, the computer device scales all gradients proportionally so that the scaled gradient norm is exactly equal to 1.0. The purpose of gradient clipping is to prevent gradient explosion from causing excessively large parameter update amplitudes and training divergence.

[0183] The computer device performs backpropagation. Backpropagation refers to calculating the partial derivative of the loss with respect to each parameter, i.e., the gradient, layer by layer, using the chain rule, starting from the loss function and following the reverse order of the computation graph. The computer device automatically completes the gradient calculation for all 28 million parameters through an automatic differentiation engine. After the calculation is completed, the computer device updates each parameter according to the update rules of the AdamW optimizer, combining the current gradient, first-order moment estimate, second-order moment estimate, and learning rate.

[0184] The computer system is configured to perform 63 batches of parameter updates per epoch. Within an epoch, the computer system sequentially performs forward propagation, loss calculation, backpropagation, and parameter updates for each batch. After all 63 batches are completed, an epoch ends. The computer system then moves to the next epoch, randomly shuffles the training subset again, and repeats the above process.

[0185] At the end of each epoch, the computer evaluates the classification performance of the current model using a validation subset. The computer feeds 1012 multimodal input features from the validation subset into the model for forward propagation, resulting in 1012 4-dimensional predicted probability vectors. The computer then calculates classification accuracy, precision, recall, and F1 score using these predicted probability vectors and their corresponding one-hot encoded supervision labels. The computer records the classification accuracy on the validation subset as the validation metric for that epoch.

[0186] The computer equipment is set to an early stopping patience value of 10, meaning that training is terminated when the classification accuracy on the validation subset does not exceed the historical best value for 10 consecutive epochs. The computer equipment also has a minimum learning rate of 0.00001; once the cosine annealing schedule reduces the learning rate to this value, it will not decrease further.

[0187] Step S84: When the loss function value meets the preset convergence condition, stop training and obtain the trained financial sentiment analysis model. The computer equipment is set with three criteria for convergence. Meeting any one of them is considered to meet the convergence criteria.

[0188] The first criterion is the convergence of the validation loss. The computer calculates the average cross-entropy loss on the validation subset after each epoch. The validation loss is considered convergent when the change in the validation loss over five consecutive epochs is less than 1e-4, that is, when the absolute value of the difference between the validation losses of two adjacent epochs is less than 0.0001.

[0189] The second criterion is validation accuracy convergence. The computer records the classification accuracy on the validation subset after each epoch. Validation accuracy is considered convergent when the change in validation accuracy over 10 consecutive epochs is less than 0.1%, meaning the absolute value of the difference between the validation accuracies of two adjacent epochs is less than 0.001.

[0190] The third criterion is reaching the maximum number of training epochs. The computer equipment is set to a maximum of 100 training epochs. When the number of training epochs reaches 100, training is forcibly stopped regardless of whether the first two criteria are met.

[0191] In addition, the computer system is equipped with an early stopping mechanism as an auxiliary convergence criterion. The early stopping mechanism is triggered when the validation accuracy does not exceed the historical best value for 10 consecutive epochs. When early stopping is triggered, the computer system immediately stops training and uses the model parameters of the epoch corresponding to the historical best validation accuracy as the final parameters.

[0192] When the preset convergence condition is met, the computer stops updating parameters and saves all parameters of the current model as the final parameters. The computer then performs a model serialization operation on the final parameters, storing all 28 million parameters in binary format as a model weight file. The model weight file contains the numerical values ​​of the parameter tensors, parameter names, parameter shapes, and corresponding layer identifier information.

[0193] The computer system performs a final evaluation of the trained financial sentiment analysis model. The evaluation method involves calculating classification accuracy, precision, recall, F1 score, and confusion matrix using a validation subset. The computer system also calculates the AUC (Area Under the Receiver Operating Characteristic) for each category, which measures the model's ability to distinguish between categories. The computer system writes the final evaluation metrics into the model's metadata file, which includes the model version number, training data size, feature dimensions, number of categories, total number of parameters, training epochs, final validation accuracy, F1 score for each category, and training timestamp.

[0194] The computer equipment deploys the trained financial public opinion analysis model to the inference service module. The inference service module receives newly collected text and image data, and after preprocessing in steps S1 to S7, generates multimodal input features, which are then fed into the trained model to perform forward propagation and output the public opinion consistency classification result.

[0195] Please see Figure 3 As shown, in one embodiment, a training device for a financial public opinion analysis model is provided, the device comprising: The acquisition module 301 is used to acquire unstructured text data and screenshots uploaded by users from the financial social platform; Extraction module 302 is used to perform semantic processing on the unstructured text data and extract text sentiment time series features; The extraction module 302 is also used to extract visual features from the screenshot to obtain image trend time series features; Alignment module 303 is used to perform spatiotemporal alignment processing on the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs; The generation module 304 is used to obtain the original text and screenshot after time alignment based on the time mapping relationship in the aligned text-image feature pair, as aligned text-image data; The generation module 304 is also used to perform multimodal depth comparison on the aligned text and image data to generate structured comparison labels; Construction module 305 is used to construct a training set based on the aligned text-image feature pairs and the structured contrast labels; Training module 306 is used to train a preset financial public opinion analysis model using the training set to obtain a trained financial public opinion analysis model.

[0196] For further details regarding the implementation of the above technical solutions by each module in the training device of the above financial public opinion analysis model, please refer to the description in the training method of the financial public opinion analysis model provided in the above application embodiments, which will not be repeated here.

[0197] It should be noted that the above-described embodiment of the device 3, when executing the training method of the financial public opinion analysis model, only illustrates the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the above functions. Furthermore, the training device for the financial public opinion analysis model and the training method embodiment of the financial public opinion analysis model provided in the above embodiment belong to the same concept, and their implementation process is detailed in the method embodiment, which will not be repeated here.

[0198] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0199] See Figure 4 The diagram shown is a schematic of a computer storage medium provided in an embodiment of this application. The computer storage medium can store multiple instructions (i.e., ... Figure 4 The computer program shown above), the instructions are adapted to be loaded and executed as described above. Figure 2 The method steps of the illustrated embodiment can be found in the following documentation for detailed execution. Figure 2 The specific details of the illustrated embodiments will not be elaborated here.

[0200] This application also provides a computer program product that stores at least one instruction, which is loaded and executed by the method for training the financial public opinion analysis model as described in the above embodiments.

[0201] Please see Figure 5 This document provides a schematic diagram of the structure of a computer device according to an embodiment of this application. Figure 5 As shown, the computer device 500 may include: at least one processor 501, at least one network interface 504, user interface 503, memory 505, and at least one communication bus 502.

[0202] The communication bus 502 is used to enable communication between these components.

[0203] The user interface 503 may include input units such as a mouse and keyboard.

[0204] The network interface 504 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0205] The processor 501 may include one or more processing cores. The processor 501 connects to various parts within the computer device 500 using various interfaces and lines, and performs various functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in memory 505, and by calling data stored in memory 505. Optionally, the processor 501 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 501 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 501 and may be implemented as a separate chip.

[0206] The memory 505 may include random access memory (RAM) or read-only memory. Optionally, the memory 505 may include a non-transitory computer-readable storage medium. The memory 505 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 505 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 505 may also be at least one storage device located remotely from the aforementioned processor 501. Figure 5 As shown, the memory 505, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0207] exist Figure 5 In the computer device 500 shown, the user interface 503 is mainly used to provide an input interface for the user and to obtain the user's input data; while the processor 501 can be used to call the application program stored in the memory 505 and specifically execute, such as Figure 2 The method shown can be referred to for details. Figure 2 As shown, it will not be elaborated further here.

[0208] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a disk, optical disk, read-only memory, or random access memory, etc.

[0209] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A training method for a financial public opinion analysis model, characterized in that, Includes the following steps: Obtain unstructured text data and user-uploaded screenshots from financial social platforms; Semantic processing is performed on the unstructured text data to extract text sentiment time series features; Visual features are extracted from the screenshot to obtain image trend time series features; The text sentiment time series features and the image trend time series features are spatiotemporally aligned to obtain aligned text-image feature pairs. Based on the time mapping relationship in the aligned text-image feature pairs, the original text and screenshot after time alignment are obtained as aligned text-image data. Multimodal depth comparison is performed on the aligned text and image data to generate structured comparison labels; A training set is constructed based on the aligned text-image feature pairs and the structured contrast labels. The pre-set financial public opinion analysis model is trained using the training set to obtain the trained financial public opinion analysis model.

2. The training method for the financial public opinion analysis model according to claim 1, characterized in that, The semantic processing steps include: The RAG technology is used to retrieve the associated contextual data of the unstructured text data, and semantic enhancement vectors are generated based on the associated contextual data as auxiliary information for semantic understanding. Simultaneously, high-frequency new words are monitored on the unstructured text data. When an unknown concept not covered in the existing financial ontology is identified, the current contextual definition of the unknown concept is obtained. The current context definition is dynamically updated to the dynamic prompt word space of the large language model in order to perform evolutionary correction on the semantics of the financial ontology in the large language model; By combining the aforementioned semantic enhancement vectors, the unstructured text data is semantically understood using an evolution-corrected large language model, thereby generating enhanced semantic information. Based on the enhanced semantic information, extract the text sentiment time-series features; The enhanced semantic information includes at least semantic vectors, entity labels, and sentiment labels after ontology evolution correction.

3. The training method for the financial public opinion analysis model according to claim 1, characterized in that, The step of performing spatiotemporal alignment processing on the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs includes: The text sentiment time series features are determined as the reference series, and the image trend time series features are determined as the series to be aligned. The minimum distance path between the reference sequence and the sequence to be aligned is calculated using a dynamic time warping algorithm. Based on the minimum distance path, correct the time offset of the image trend time series features; Based on the corrected time offset, the screenshot is flexibly mapped to its corresponding actual time point to obtain aligned text and image data; Based on the time mapping relationship in the aligned text-image feature pair, the original text and screenshot after time alignment are obtained as the aligned text-image data.

4. The training method for the financial public opinion analysis model according to claim 3, characterized in that, The step of performing multimodal deep comparison on the aligned text and image data to generate structured comparison labels includes: The aligned text and image data are then input into the multimodal large model; Using the multimodal large model, subjective investor opinions are extracted from the text data in the aligned text and image data, and objective profit and loss facts are extracted from the image data. Based on the investor's subjective statement and the objective profit and loss facts, a structured comparison label is generated. The structured comparison label is used to characterize the consistency relationship between the investor's subjective statement and the objective profit and loss facts.

5. The training method for the financial public opinion analysis model according to claim 1, characterized in that, The step of constructing a training set based on the aligned text-image feature pairs and the structured contrast labels includes: The text sentiment time series features and image trend time series features in the aligned text-image feature pair are fused to form multimodal input features; The structured contrast labels are converted into classification labels, and the classification labels are used as supervision labels for the training samples. The multimodal input features and the supervision labels are used to construct training samples, which are then aggregated to form the training set.

6. The training method for the financial public opinion analysis model according to claim 1 or 5, characterized in that, The financial public opinion analysis model is a classifier based on a multimodal neural network; The input layer of the classifier is used to receive the multimodal input features; The output layer of the classifier is used to output the public opinion consistency classification result.

7. The training method for the financial public opinion analysis model according to claim 1 or 6, characterized in that, The step of training a pre-defined financial public opinion analysis model using the training set to obtain a trained financial public opinion analysis model includes: The multimodal input features are input into the financial public opinion analysis model to obtain predicted labels; Calculate the loss function value between the predicted label and the corresponding supervised label in the training set; With the goal of minimizing the loss function value, the model parameters of the financial public opinion analysis model are iteratively optimized; When the loss function value meets the preset convergence condition, training stops, and the trained financial public opinion analysis model is obtained.

8. A training device for a financial public opinion analysis model, characterized in that, include: The acquisition module is used to acquire unstructured text data and screenshots uploaded by users from financial social platforms; The extraction module is used to perform semantic processing on the unstructured text data and extract the text sentiment time series features; The extraction module is also used to extract visual features from the screenshot to obtain image trend time series features; The alignment module is used to perform spatiotemporal alignment processing on the text sentiment time series features and the image trend time series features to obtain aligned text-image feature pairs. The generation module is used to obtain the original text and screenshot after time alignment based on the time mapping relationship in the aligned text-image feature pair, as aligned text-image data; The generation module is also used to perform multimodal depth comparison on the aligned text and image data to generate structured comparison labels; The construction module is used to construct a training set based on the aligned text-image feature pairs and the structured contrast labels; The training module is used to train the preset financial public opinion analysis model using the training set to obtain the trained financial public opinion analysis model.

9. A computer device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the training method for the financial sentiment analysis model as described in any one of claims 1 to 7.

Images