Financial news data cleaning system and method

Abstract

The application relates to the technical field of data processing, and discloses a financial news data cleaning system, which comprises a data normalization preprocessing module, a semantic splitting and structuring module, a label checking and cleaning module, a double-path vector representation module and a semantic deduplication filtering module. The data normalization preprocessing module is used for normalizing and preprocessing multi-source heterogeneous financial news original data to obtain standardized financial news data. The semantic splitting and structuring module is used for performing semantic structuring processing on a single-theme sub-article set to obtain structured financial news data. The label checking and cleaning module is used for performing label cleaning processing on the structured financial news data to obtain target financial news data. The double-path vector representation module is used for performing vector fusion on dense vectors and sparse vectors to obtain double-path vector representation data. The semantic deduplication filtering module is used for performing deduplication filtering processing on the target financial news data based on a semantic similarity score to obtain a cleaned financial news data set. The application can improve the accuracy of financial news data cleaning.

Description

Technical Field

[0001] This invention relates to a financial news data cleaning system and method, belonging to the field of data processing technology. Background Technology

[0002] With the rapid development of financial information services, multi-source and heterogeneous financial news data has experienced explosive growth. Existing financial news data cleaning methods mainly employ rules based on Uniform Resource Locator (URL) exact matching or title string similarity for deduplication, relying on manual annotation or simple keyword extraction to generate stock and theme tags. However, this method is only suitable for single-source or well-formatted news texts, and is completely unable to handle the complex scenarios in financial news where a single report involves multiple stocks or themes from various sources such as selected announcements, research reports, interpretations of major price surges, hot news, and institutional chief analysts' opinions. In actual business, the data producers of different sources vary, the schema of extended fields lacks a unified standard, and there is a large amount of semantic repetition in reports on the same financial event from different sources. Traditional methods, lacking the ability to understand the semantics of multi-topic content, cannot accurately segment the data, resulting in mixed tags, inaccurate semantic vector representations, and coarse-grained push notifications.

[0003] This defect causes downstream systems to either miss detections due to their inability to recognize synonyms or generate a large number of duplicate pushes due to reports of the same event across different sources, severely impacting user experience. Summary of the Invention

[0004] This invention provides a financial news data cleaning system, the main purpose of which is to improve the accuracy of financial news data cleaning.

[0005] To achieve the above objectives, the present invention provides a financial news data cleaning system, the system comprising: The data normalization and preprocessing module is used to acquire multi-source heterogeneous financial news raw data, read stored financial news data, and perform normalization and preprocessing on the multi-source heterogeneous financial news raw data to obtain standardized financial news data. The semantic segmentation and structuring module is used to call a preset large language model agent to perform multi-topic semantic segmentation on the standardized financial news data, obtain a set of single-topic sub-articles, and perform semantic structuring processing on the set of single-topic sub-articles to obtain structured financial news data. The tag verification and cleaning module is used to match and verify the tags in the structured financial news data with the tag master data to obtain valid tags, so as to perform tag cleaning processing on the structured financial news data to obtain the target financial news data. The dual-path vector representation module is used to perform dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors, and to perform vector fusion on the dense vectors and the sparse vectors to obtain dual-path vector representation data. The semantic deduplication filtering module is used to perform semantic similarity retrieval in the financial news data based on the dual-path vector representation data, obtain a semantic similarity score, and perform deduplication filtering on the target financial news data based on the semantic similarity score to obtain a cleaned financial news dataset.

[0006] Optionally, the standardization preprocessing of the multi-source heterogeneous raw financial news data to obtain standardized financial news data includes: The multi-source heterogeneous financial news raw data is preprocessed using general rules to obtain general preprocessed data; Extract the source identifiers carried by the multi-source heterogeneous financial news raw data from the general preprocessed data; Based on the source identifier, the general preprocessed data is parsed with differentiated extended fields to obtain candidate label prompts and source structured fields; Based on the candidate tag suggestions and the source structured fields, the general preprocessed data is adapted to the source and normalized in format to obtain standardized financial news data.

[0007] Optionally, based on the source identifier, the general preprocessed data is parsed using differentiated extended fields to obtain candidate label hints and source structured fields, including: When the source identifier matches the selected source of the announcement, the stock code array is parsed from the extended field, verified with the stock master data and used as a candidate tag suggestion, and the influence, summary and announcement type fields are extracted as source structure fields; When the source identifier matches the research report source, the research report page number and tag list are parsed from the extended field, and the stock tag is suggested as a candidate tag after verification by the stock master data. When the source identifier matches the source of the surge interpretation, the subject name, subject driving logic and the number of stocks rising in the subject are parsed from the extended fields. The subject list is used as a candidate tag prompt and the subject details are used as the source structure field. When the source identifier matches a trending news source, the topic, stock, sorting index, and tag time are parsed from the extended fields, and the stored data can be queried by URL to determine whether there are duplicates. When the source identifier matches the chief opinion source of an organization, the tag list is parsed from the extended field, the source type identifier is set, and the knowledge planet jump address is fixed as the source structure field; When the source identifier does not match any configured dedicated processing logic, the word filtering service is invoked to extract stock entities mentioned in the article text based on financial dictionaries and regular expression rules. The extraction results are then verified against the stock master data and used as candidate tag suggestions.

[0008] Optionally, the step of invoking a preset large language model agent to perform multi-topic semantic segmentation on the standardized financial news data to obtain a set of single-topic sub-articles, including: Standardized financial news data is segmented into candidate semantic units at the paragraph and sentence group granularity, and topic segmentation prompt words are injected into the large language model agent. The agent labels the topic category to which each semantic unit belongs and the topic switching points between adjacent units. Based on the topic switching point, the candidate semantic units are aggregated into several candidate topic clusters, and the entity co-occurrence density and financial element coverage of each candidate topic cluster are statistically analyzed. If the co-occurrence density of entities within the same candidate topic cluster is lower than the preset agglomeration threshold, then backtrack and split the cluster, and merge the low-agglomeration fragments into the adjacent topic clusters; If the similarity of financial element coverage of adjacent candidate topic clusters is higher than the preset separation threshold, they will be merged into the same single topic sub-article. The candidate topic clusters, after being refined through cohesion and separation, are completed into independent single-topic sub-articles, resulting in a set of single-topic sub-articles.

[0009] Optionally, performing semantic structuring processing on the single-topic sub-article set to obtain structured financial news data includes: Extract the title, body text, abstract, and impact fields from the single-topic sub-article collection; The summary field and the influence field are normalized to obtain a content summary and sentiment polarity identifier; The extracted title, body text, content summary, and sentiment polarity identifier are encapsulated according to a preset field structure to obtain structured financial news data.

[0010] Optionally, the step of matching and validating the tags in the structured financial news data with the tag master data to obtain valid tags includes: Candidate tags in the structured financial news data are sorted by tag type to obtain a set of branched tags; Extract the tag codes from the branch tag set and query and match them with the stock index and theme index pre-stored in the tag master data to obtain preliminary matching tags; Metadata appending and invalid item removal are performed on the initially matched tags to obtain valid tags.

[0011] Optionally, the step of performing tag cleaning on the structured financial news data based on the effective tags to obtain the target financial news data includes: Perform a deduplication operation on the valid tags according to the tag code to obtain a deduplicated tag set; The sub-articles with empty duplicate tag sets are removed from the structured financial news data to obtain the retained sub-article set; The retained sub-article set and the corresponding deduplication tag set are assembled into a structured document with fixed key names to obtain the target financial news data.

[0012] Optionally, the step of performing dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors includes: Semantic embedding is performed on each article element of the target financial news data to obtain a dense embedding vector; Based on preset weights, the dense embedding vectors of each dimension are weighted and synthesized to obtain a dense vector; The article elements of the target financial news data are subjected to term activation encoding to obtain an initial sparse vector; The dimensional components in the initial sparse vector are subjected to quality filtering to obtain a sparse vector.

[0013] Optionally, the step of fusing the dense vector and the sparse vector to obtain dual-path vector representation data includes: The dense vector is serialized according to a floating-point array format of a preset dimension to obtain a dense vector field; The non-zero dimension components in the sparse vector are extracted into a set of key-value pairs of term identifiers and weight values ​​to obtain the sparse vector field. The dense vector field and the sparse vector field are encapsulated into a unified data structure containing dual-path representation fields to obtain dual-path vector representation data.

[0014] A method for cleaning financial news data, characterized in that the method includes: Acquire raw financial news data from multiple sources and read stored financial news data. Perform normalization preprocessing on the raw financial news data from multiple sources and obtain standardized financial news data. The standardized financial news data is subjected to multi-topic semantic segmentation by a pre-defined large language model intelligent agent to obtain a set of single-topic sub-articles, and the single-topic sub-articles are subjected to semantic structuring processing to obtain structured financial news data. The tags in the structured financial news data are matched and verified with the tag master data to obtain valid tags, and then the structured financial news data is cleaned to obtain the target financial news data. Dense vector encoding and sparse vector encoding are performed on the article elements of the target financial news data to obtain dense vectors and sparse vectors. The dense vectors and sparse vectors are then fused to obtain dual-path vector representation data. Based on the dual-path vector representation data, semantic similarity retrieval is performed on the financial news data to obtain a semantic similarity score. Based on the semantic similarity score, the target financial news data is subjected to deduplication filtering to obtain a cleaned financial news dataset.

[0015] This invention, through standardized preprocessing of the multi-source heterogeneous financial news raw data, helps eliminate the heterogeneity of different sources in data format, extended fields, and content quality. This provides a unified input standard for subsequent large language model intelligent agent processing and ensures a reliable historical comparison benchmark in the semantic deduplication stage. By invoking a pre-set large language model intelligent agent to perform multi-topic semantic segmentation on the standardized financial news data, this invention effectively avoids label ambiguity and inaccurate semantic vector representation caused by multi-topic mixing, thereby obtaining the single-topic sub-article set. Furthermore, by matching and verifying the labels in the structured financial news data with the label master data, this invention can intercept phantom labels and outdated labels generated during the extraction process of the large language model intelligent agent, thus ensuring... The semantic consistency between financial news data and real-time market data systems enhances the trust of downstream business systems in tagged data. This invention utilizes dense and sparse vector encoding on the article elements of the target financial news data, which helps to balance deep semantic similarity capture and precise keyword matching capabilities. This provides a foundation for semantic-level deduplication that balances semantic equivalence and keyword accuracy, solving the semantic drift problem inherent in single dense vectors in the financial field. By performing semantic similarity retrieval on the financial news data based on the dual-path vector representation data, this invention can overcome the limitations of Uniform Resource Locators (URLs) and exact title text matching, accurately identifying semantically repetitive content describing the same financial event from news published by different sources, thereby expanding the deduplication coverage and reducing the false positive rate. Therefore, this invention can improve the accuracy of financial news data cleaning. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of the modules used to implement the financial news data cleaning system according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the architecture of a financial news data cleaning system provided in an embodiment of the present invention; Figure 3A flowchart illustrating a financial news data cleaning method according to an embodiment of the present invention; Figure 4 A schematic diagram of a computer device for a financial news data cleaning system provided in an embodiment of the present invention; The objectives, features, and advantages of this invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] This application provides a financial news data cleaning system. The executing entity of the financial news data cleaning system includes, but is not limited to, at least one of the following electronic devices that can be configured to execute the method provided in this application: a server, a terminal, etc. In other words, the financial news data cleaning system can be executed by software or hardware installed on a terminal device or a server device. The server includes, but is not limited to, a single server, a server cluster, a cloud server, or a cloud server cluster.

[0019] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments.

[0020] like Figure 1 The diagram shown is a schematic representation of the modules of the financial news data cleaning system of the present invention.

[0021] The financial news data cleaning system 100 of this invention can be installed in an electronic device. Depending on its functions, the financial news data cleaning system includes a data normalization preprocessing module 101, a semantic segmentation and structuring module 102, a tag verification and cleaning module 103, a dual-path vector representation module 104, and a semantic deduplication filtering module 105. The modules described in this invention can also be referred to as units, which are a series of computer program segments that can be executed by the processor of an electronic device and perform a fixed function, stored in the memory of the electronic device.

[0022] like Figure 2 The diagram shown is an architectural schematic of a financial news data cleaning system provided in an embodiment of the present invention: In the diagram, the overall system architecture is divided into three layers from bottom to top: data preprocessing and message adaptation layer, AI intelligent cleaning core engine layer, and vectorized storage and push layer.

[0023] The data preprocessing and message adaptation layer is responsible for performing general rule preprocessing and source differentiation parsing on multi-source heterogeneous raw data from the results database, market master data, word filtering service and management console, extracting candidate tags and outputting standardized data; The results database (db_info_result, divided into monthly tables) stores high-quality structured financial information data accumulated through historical cleaning processes. It serves as a benchmark data source for incremental comparison, duplicate detection, and knowledge backtracking, and supports horizontal expansion of storage by month. The MarketDataServer provides real-time market snapshots, historical market sequences, standardized stock code mapping tables, and hierarchical relationships of themes and concepts, providing authoritative references for subsequent stock tag extraction, theme association, and master data verification, ensuring accurate correspondence between tags and current market entities. The WordFilterServer, based on a sensitive word library, industry blacklists, and a regular expression rule engine, performs initial screening and compliance verification on raw text, blocking advertisements, spam, and illegal content, reducing processing noise in downstream AI engines, and improving input data quality. The management console (RELOAD / RELOADALL) serves as the central hub for system operation and strategy configuration, supporting hot loading of the sensitive word library, dynamic refreshing of cleaning strategies, service status monitoring, and batch reloading of global parameters. This enables configuration changes and canary releases without downtime, ensuring continuous and stable system operation.

[0024] The core engine layer of AI-powered intelligent cleaning uses an LLM Agent cluster (including intelligent tag extraction, topic / stock extractor, sentiment tendency determination, etc.) to perform multi-topic semantic segmentation and structured processing on standardized data, generating structured financial news data containing titles, body text, summaries, sentiment polarity identifiers, and candidate tags. The vectorized storage and push layer sequentially performs tag verification and cleaning, dense vector encoding and sparse vector encoding, and dual-path vector fusion on the structured data to obtain dual-path vector representation data. Based on this data, semantic similarity retrieval and deduplication filtering are performed in the Elasticsearch cluster. Finally, the cleaned high-quality documents are written into the index and pushed to the downstream business system.

[0025] This diagram clearly illustrates the five-layer pipeline architecture proposed in this invention: "preprocessing → intelligent splitting → tag verification → dual-path encoding → semantic deduplication," providing a system-level view for the functional implementation of subsequent modules.

[0026] In this embodiment of the invention, the functions of each module / unit are as follows: The data normalization preprocessing module 101 is used to acquire multi-source heterogeneous financial news raw data, read stored financial news data, and perform normalization preprocessing on the multi-source heterogeneous financial news raw data to obtain standardized financial news data.

[0027] This invention helps eliminate the heterogeneity of different information sources in terms of data format, extended fields, and content quality by performing standardized preprocessing on the original multi-source heterogeneous financial news data. This provides a unified input standard for subsequent large language model intelligent agent processing and ensures that the semantic deduplication stage has a reliable historical comparison benchmark.

[0028] The multi-source heterogeneous financial news raw data includes data from at least one source among temporary information databases, selected announcements, research reports, interpretations of major price surges, hot news, and chief institutional opinions.

[0029] Specifically, the standardization preprocessing of the multi-source heterogeneous raw financial news data to obtain standardized financial news data includes: The multi-source heterogeneous financial news raw data is preprocessed using general rules to obtain general preprocessed data; Extract the source identifiers carried by the multi-source heterogeneous financial news raw data from the general preprocessed data; Based on the source identifier, the general preprocessed data is parsed with differentiated extended fields to obtain candidate label prompts and source structured fields; Based on the candidate tag suggestions and the source structured fields, the general preprocessed data is adapted to the source and normalized in format to obtain standardized financial news data.

[0030] The general rule preprocessing refers to a cleansing operation, independent of the source, uniformly executed on various information sources. The source identifier is a unique number used to distinguish news source channels. The candidate tag prompt is a set of high-confidence tags parsed from the extended fields to guide subsequent intelligent modules in tag extraction. The source structured field is structured information extracted from the unique extended fields of each information source, including topic name, driving logic, research report page number, or sorting index. The format normalization refers to the process of unifying the outputs of different information sources into a standard field structure and data type.

[0031] It should be explained that, since the data producers of different information sources are different, the schema of extended fields lacks a unified standard. If a unified parsing is performed directly on all data, it will result in incomplete extraction of structured information or noise interference. Therefore, it is necessary to first perform preprocessing with general rules unrelated to the information source to eliminate basic noise, and then extract the information source identifier from the cleaned data. Based on this identifier, differentiated parsing is then performed to balance data cleanliness and the integrity of information source-specific information.

[0032] Furthermore, based on the source identifier, the general preprocessed data is parsed using differentiated extended fields to obtain candidate label hints and source structured fields, including: When the source identifier matches the selected source of the announcement, the stock code array is parsed from the extended field, verified with the stock master data and used as a candidate tag suggestion, and the influence, summary and announcement type fields are extracted as source structure fields; When the source identifier matches the research report source, the research report page number and tag list are parsed from the extended field, and the stock tag is suggested as a candidate tag after verification by the stock master data. When the source identifier matches the source of the surge interpretation, the subject name, subject driving logic and the number of stocks rising in the subject are parsed from the extended fields. The subject list is used as a candidate tag prompt and the subject details are used as the source structure field. When the source identifier matches a trending news source, the topic, stock, sorting index, and tag time are parsed from the extended fields, and the stored data can be queried by URL to determine whether there are duplicates. When the source identifier matches the chief opinion source of an organization, the tag list is parsed from the extended field, the source type identifier is set, and the knowledge planet jump address is fixed as the source structure field; When the source identifier does not match any configured dedicated processing logic, the word filtering service is invoked to extract stock entities mentioned in the article text based on financial dictionaries and regular expression rules. The extraction results are then verified against the stock master data and used as candidate tag suggestions.

[0033] The stock master data refers to the authoritative basic data set containing stock codes, names, markets, and security types obtained from the market data system (MarketDataServer). The word filtering service (WordFilterServer) refers to a backend microservice that quickly extracts stock codes or abbreviations from unstructured text based on a professional financial dictionary and regular expression matching rules. The sorting index refers to a numerical field that marks the order of articles in a news collection on the same topic. The redirect address refers to the URL used to link to a third-party content platform when displayed on the front end.

[0034] It should be explained that the selected announcements and research reports, whose data comes from exchange disclosures or official releases by securities firms, have stock codes in their extended fields with high prior accuracy. Therefore, a direct path of "parsing → master data verification → direct use as candidate tag prompts" is adopted to reduce the inference overhead of subsequent intelligent modules. The core value of the surge interpretation sources lies in the interpretation of thematic content, so thematic information is parsed first as a guiding clue for subsequent article segmentation. Hot news sources, due to their high-frequency update characteristics, rely on URL queries to check the stored data to determine whether the news already exists and update the sorting index to avoid duplicate processing. For unknown sources without dedicated logic, the system is downgraded to a word filtering service for rule-based candidate tag extraction to ensure the robustness and scalability of the system. Adding a new source only requires registering the corresponding parser to access it, without modifying the core preprocessing process.

[0035] The semantic segmentation and structuring module 102 is used to call a preset large language model agent to perform multi-topic semantic segmentation on the standardized financial news data, obtain a single-topic sub-article set, and perform semantic structuring processing on the single-topic sub-article set to obtain structured financial news data.

[0036] This invention utilizes a pre-defined large language model agent to perform multi-topic semantic segmentation on standardized financial news data. This effectively avoids the problems of label ambiguity and inaccurate semantic vector representation caused by the mixing of multiple topics, thereby obtaining the set of single-topic sub-articles. The pre-defined large language model agent refers to a natural language processing service based on a large-scale language model deployed in the background. It is configured to receive input prompts containing task instructions, Few-shot examples, and JSON output format constraints, identify the topic boundaries in the article, and split the original news with mixed multiple topics into multiple single-topic sub-article objects containing independent titles, body text, summaries, and tag lists. The set of single-topic sub-articles refers to a collection of several sub-articles obtained after semantic segmentation of an original news article containing multiple investment topics. Each sub-article discusses only a single topic or a single stock, and each contains an independent title, body text, summary, and tag list.

[0037] In detail, the process involves invoking a pre-defined large language model agent to perform multi-topic semantic segmentation on the standardized financial news data, resulting in a set of single-topic sub-articles, including: Standardized financial news data is segmented into candidate semantic units at the paragraph and sentence group granularity, and topic segmentation prompt words are injected into the large language model agent. The agent labels the topic category to which each semantic unit belongs and the topic switching points between adjacent units. Based on the topic switching point, the candidate semantic units are aggregated into several candidate topic clusters, and the entity co-occurrence density and financial element coverage of each candidate topic cluster are statistically analyzed. If the co-occurrence density of entities within the same candidate topic cluster is lower than the preset agglomeration threshold, then backtrack and split the cluster, and merge the low-agglomeration fragments into the adjacent topic clusters; If the similarity of financial element coverage of adjacent candidate topic clusters is higher than the preset separation threshold, they will be merged into the same single topic sub-article. The candidate topic clusters, after being refined through cohesion and separation, are completed into independent single-topic sub-articles, resulting in a set of single-topic sub-articles.

[0038] Here, the candidate semantic unit refers to a text fragment of atomic granularity obtained by doubly segmenting the main text content of standardized financial news data according to paragraph boundaries and sentence-ending punctuation (period, semicolon, question mark), with each unit carrying a relatively complete semantic statement; the topic category refers to the classification label marked by the large language model agent for each semantic unit, with the value being a string identifier of a stock name, theme name, or event type actually involved in the article; the topic switching point refers to the boundary position index where the topic category changes between adjacent semantic units, corresponding to the turning point where the narrative focus in the article shifts from one investment topic to another; the candidate topic cluster refers to a text block aggregated from consecutive semantic units under the same topic category, representing the initial set of discussions on a specific investment topic in the article; the entity co-occurrence density refers to the density of entities appearing within a candidate topic cluster. The ratio of the deduplication count of financial entities such as different stock codes, theme names, and financial indicator keywords to the total number of words in the theme cluster is used to quantify the theme focus of the cluster. The financial element coverage refers to the measure of the completeness of the types of financial semantic elements contained in a candidate theme cluster. The financial semantic elements include at least three of the following: stock code mentions, theme sector associations, words indicating the direction of rise and fall, amount / proportion values, and time node information. The higher the coverage, the more comprehensive the investment decision information carried by the cluster. The preset cohesion threshold is the lower limit of entity co-occurrence density that can ensure the minimum readability and information content of a single theme sub-article, obtained from the statistics of historical annotated corpus. The preset separation threshold is the upper limit of similarity set according to the empirical distribution of the overlap of financial elements between different theme clusters in news of the same theme. If the value is exceeded, it is considered that the two clusters are actually describing the same investment event.

[0039] Furthermore, a context-aware sliding window annotation method based on a large language model agent can be used to identify the topic category of each semantic unit and the topic switching point between adjacent units. That is, by feeding the text of standardized financial news data into the large language model agent in a fixed window size (3 sentences by default), the agent annotates the topic of each sentence in the current window at each step and outputs a binary judgment result on whether there is a topic switch at the right boundary of the window. In this way, the topic segmentation and switching point location of the whole text are completed in a streaming manner window by window. Furthermore, the statistical analysis of entity co-occurrence density and financial element coverage for each candidate topic cluster specifically includes: traversing each semantic unit in the candidate topic cluster, using a financial entity recognition dictionary and regular expression pattern matching to extract stock codes, topic names, price fluctuation direction words, monetary values, and time nodes; counting the total number of duplicate entities and calculating the ratio of this number to the total number of words in the topic cluster to obtain the entity co-occurrence density; simultaneously, checking whether the cluster contains at least three of the five financial semantic elements: stock code mentions, topic sector associations, price fluctuation direction judgment words, monetary or proportional values, and time node information; dividing the number of matched element types by the total number of element types to obtain the financial element coverage score. Furthermore, if the entity co-occurrence density within the same candidate topic cluster is lower than a preset cohesion threshold, then backtracking and splitting the low-cohesion fragments into adjacent topic clusters specifically includes: when the entity co-occurrence density of a candidate topic cluster is lower than the preset cohesion threshold, the cluster is split into smaller sub-fragments in reverse along the topic switching point, and the entity Jaccard similarity between each sub-fragment and the adjacent clusters on the left and right is calculated one by one. The sub-fragments with higher similarity are migrated as a whole to the corresponding adjacent clusters until the entity co-occurrence density of the cluster reaches or exceeds the preset cohesion threshold, thereby eliminating the semantic dilution phenomenon within the topic cluster; Furthermore, if the similarity of the financial element coverage of adjacent candidate topic clusters is higher than a preset separation threshold, they are merged into the same single topic sub-article. Specifically, this includes: calculating the cosine similarity of the financial element coverage vectors of two adjacent candidate topic clusters; when the similarity exceeds the preset separation threshold, the semantic units of the two clusters are sequentially concatenated into a merged cluster and given a unified topic category identifier; at the same time, the source information and parent_id of the original cluster are recorded in the metadata of the merged cluster for subsequent tracing. Furthermore, the step of completing the candidate topic clusters after agglomeration and separation into independent single-topic sub-articles specifically includes: for each corrected candidate topic cluster, generating a sub-article title based on the content of its first semantic unit, concatenating the main text content of all semantic units within the cluster in the original order to form the sub-article body, calling the large language model agent to generate a summary text based on the body content that does not exceed a preset word limit, and directly inheriting the tags from the known candidate tags passed in during the preprocessing stage that match the current topic category as the tag list of the sub-article, and finally encapsulating them into an independent structured sub-article object containing a title, body, summary, tag list, influence score, and classification field. The set of all sub-article objects constitutes the single-topic sub-article set.

[0040] This invention eliminates field differences between multi-topic sub-articles by performing semantic structuring on the single-topic sub-article set, ensuring that each sub-article has a complete title, body text, summary, and sentiment marker, thereby improving the standardization and consistency of the data. The structured financial news data refers to a financial news data set with a unified data format obtained by encapsulating the title, body text, content summary, and sentiment polarity marker in the single-topic sub-article set according to a preset field structure.

[0041] Specifically, the semantic structuring processing performed on the single-topic sub-article set to obtain structured financial news data includes: Extract the title, body text, abstract, and impact fields from the single-topic sub-article collection; The summary field and the influence field are normalized to obtain a content summary and sentiment polarity identifier; The extracted title, body text, content summary, and sentiment polarity identifier are encapsulated according to a preset field structure to obtain structured financial news data.

[0042] Wherein, the title refers to the title text of a single-topic sub-article, used to summarize the core theme of the sub-article; the body text refers to the main argument of the single-topic sub-article excluding the title and summary; the summary field refers to a brief summary text of the body text in the sub-article, which can be empty or the same as the title; the influence field refers to a level identifier parsed from the extended fields in the preprocessing stage, used to characterize the potential impact of the news on market sentiment; the content summary refers to a non-empty, brief descriptive text that is different from the title, obtained after normalizing the summary field; the sentiment polarity identifier refers to a positive, neutral, or negative level label obtained after normalizing the influence field; and the preset field structure refers to a predefined data pattern containing fixed key names and corresponding data types.

[0043] Furthermore, a joint normalization method based on null value truncation and default value mapping can be used to normalize the summary field and the influence field to obtain a content summary and a sentiment polarity identifier. Specifically, this includes: when the summary field is empty, the first 100 characters of the main text are truncated as a substitute summary text; when the summary field is the same as the title content, the first 100 characters of the main text are also truncated and replaced, and the replaced text is recorded as the content summary; when the influence field has a valid value, the value is directly read as the sentiment polarity identifier. The sentiment polarity identifier refers to the sentiment level mark obtained after normalizing the influence field, which is encoded as positive, neutral, or negative according to the convention. When the judgment fails or the field is missing, the default value is assigned to neutral. Furthermore, the extracted title, body text, content summary, and sentiment polarity identifier can be encapsulated according to a preset field structure using a key-value alignment-based serialization encapsulation method to obtain structured financial news data. The specific steps are as follows: First, create an empty dictionary object containing four preset key names: title, body text, summary, and influence. Second, assign the extracted title text to the title key, the body text to the body key, the content summary to the summary key, and the sentiment polarity identifier to the influence key. Then, validate the data type of each key-value pair in the dictionary to ensure that the text content is all of string type and the influence is an integer value. Finally, serialize the validated dictionary object into a JSON format string to obtain the structured financial news data.

[0044] The tag verification and cleaning module 103 is used to match and verify the tags in the structured financial news data with the tag master data to obtain valid tags, so as to perform tag cleaning processing on the structured financial news data to obtain target financial news data.

[0045] This invention intercepts illusory and outdated tags generated during the extraction process of large language model agents by matching and verifying the tags in the structured financial news data with the tag master data. This ensures the semantic consistency between financial news data and real-time market data systems and enhances the trust of downstream business systems in the tag data. The effective tags refer to tags that are confirmed to exist after secondary verification of stock master data or theme master data and that match the actual listed trading targets or active theme sectors in the current financial market.

[0046] Specifically, the process of matching and validating the tags in the structured financial news data with the tag master data to obtain valid tags includes: Candidate tags in the structured financial news data are sorted by tag type to obtain a set of branched tags; Extract the tag codes from the branch tag set and query and match them with the stock index and theme index pre-stored in the tag master data to obtain preliminary matching tags; Metadata appending and invalid item removal are performed on the initially matched tags to obtain valid tags.

[0047] The candidate tags refer to the set of tags to be verified after semantic splitting and structured encapsulation, including known candidate tags passed through from the preprocessing stage and tags inherited from the splitting stage; the tag type refers to the enumeration value used to distinguish the entity category to which the tag belongs, including stock type and theme type; the branched tag set refers to the set formed by sorting candidate tags according to tag type, corresponding to stock tag queues and theme tag queues respectively; the stock index refers to a hash table structure built from the stock master data (ShtType=2 / 6 / 47 / 80 / 13 / 27) loaded from the market data system, with stock code as the key and associated with stock name and listing market; the theme index refers to a hash table structure built from the theme master data (ShtType=86) loaded from the market data system, with theme code as the key and associated with theme name and sector information; the preliminary matching tag refers to the tag record that is successfully matched in the stock index or theme index; the invalid item refers to the tag that fails to match in the index or is missing key metadata fields after matching.

[0048] Furthermore, a dual-path sorting method based on tag type routing can be used to sort the candidate tags according to tag type, resulting in a set of branched tags. For example, each tag in the candidate tag list is traversed, and its tag type field is read. When the field value is stock type, the code is assigned to the first queue; when the field value is theme type, the code is assigned to the second queue. The first and second queues together constitute the set of branched tags. The tag codes in the set of branched tags can be extracted using a fast lookup and matching method based on hash indexes. These codes are then queried and matched with the stock index and theme index pre-stored in the tag master data to obtain preliminary matching tags. For example, stock master data (ShtType = 2 / 6 / 47 / 80 / 13 / 27) is loaded from the market data system (MarketDataServer) to build a stock index, and theme master data (ShtType = 86) is loaded to build a theme index. The first queue is traversed... For each stock code in the column, an exact match is performed in the stock index. If a match is found, the corresponding stock name and listing market are appended to the tag record. For each theme code in the second queue, an exact match is performed in the theme index. If a match is found, the corresponding theme name and sector information are appended to the tag record. A filtering method based on metadata completeness verification can be used to append metadata and remove invalid items from the initially matched tags to obtain valid tags. The specific steps are as follows: First, check whether each initially matched tag has both a code field and a name field. For tags that only have a code but lack a name, attempt to complete the name from the master data. Second, discard tags that still cannot obtain a valid name after completion, as well as tags that did not generate any match during index traversal. Finally, re-aggregate all remaining valid tags according to their respective sub-article dimensions to obtain the final set of valid tags.

[0049] This invention eliminates data redundancy of duplicate tags within the same sub-article by performing tag cleaning on the structured financial news data, and intercepts empty-tag sub-articles without any valid tags from entering the downstream processing flow, thereby reducing index storage overhead and improving tag-based retrieval accuracy. The target financial news data refers to a set of structured financial news data that, after tag deduplication and tagless filtering, has at least one valid tag for each sub-article and a unique tag code, and serves as the input data for the downstream dual-path vector representation module.

[0050] Specifically, the step of performing tag cleaning on the structured financial news data based on the effective tags to obtain the target financial news data includes: Perform a deduplication operation on the valid tags according to the tag code to obtain a deduplicated tag set; The sub-articles with empty duplicate tag sets are removed from the structured financial news data to obtain the retained sub-article set; The retained sub-article set and the corresponding deduplication tag set are assembled into a structured document with fixed key names to obtain the target financial news data.

[0051] The deduplication tag set refers to the final tag combination within the same sub-article, after the tag code is uniquely processed, with each tag code retaining only one record. The retained sub-article set refers to the set of sub-articles that remain after removing sub-articles from the structured financial news data whose deduplication tag set is empty, and which carry at least one valid tag.

[0052] Furthermore, the valid tags can be deduplicated according to their tag codes using a unique deduplication method based on sequential traversal to obtain a set of deduplicated tags. For example, each valid tag in the same sub-article can be traversed one by one, and the tag code of each tag can be compared with the recorded code list. If the code does not exist in the code list, it can be added to the list and the corresponding tag can be kept. If it already exists, the tag can be discarded directly. After the traversal is completed, the tags in the list are the set of deduplicated tags. The sub-articles with empty deduplication tag sets are removed from the structured financial news data by a threshold filtering method based on the number of elements, resulting in a set of retained sub-articles. For example, the number of elements contained in the deduplication tag set corresponding to each sub-article in the structured financial news data is checked one by one. When the number of elements is zero, the sub-article is deleted from the dataset. When the number of elements is greater than zero, it is included in the set of retained sub-articles. The target financial news data is obtained by assembling the set of retained sub-articles and the corresponding set of deduplication tags into a structured document with fixed key names using a document assembly method based on fixed key name mapping. The specific steps are as follows: First, a data structure containing several predetermined key names is established for each sub-article in the set of retained sub-articles; second, the set of deduplication tags is assigned to the tag keys, and the original content of the sub-article is assigned to the corresponding keys; finally, the data structure is converted into a structured document format to obtain the target financial news data.

[0053] The dual-path vector representation module 104 is used to perform dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors, and to perform vector fusion on the dense vectors and sparse vectors based on preset weights to obtain dual-path vector representation data.

[0054] This invention utilizes dense and sparse vector encoding on the article elements of the target financial news data. This approach helps to balance deep semantic similarity capture with precise keyword matching, thus providing a foundation for semantic-level deduplication that balances semantic equivalence and keyword accuracy. It also addresses the semantic drift problem inherent in single dense vectors in the financial field. The dense vector is a unified vector representation obtained by weighting and summing the dense embedding vectors corresponding to each article element according to preset weights and then normalizing them using the L2 norm. The sparse vector is a set of key-value mappings formed by quality filtering of the initial high-dimensional sparse representation obtained after term activation encoding and retaining the effective non-zero dimension components and their corresponding weights.

[0055] Specifically, the process of performing dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors includes: Semantic embedding is performed on each article element of the target financial news data to obtain a dense embedding vector; Based on preset weights, the dense embedding vectors of each dimension are weighted and synthesized to obtain a dense vector; The article elements of the target financial news data are subjected to term activation encoding to obtain an initial sparse vector; The dimensional components in the initial sparse vector are subjected to quality filtering to obtain a sparse vector.

[0056] Wherein, the dense embedding vector refers to a fixed-dimensional high-dimensional continuous vector representation obtained by encoding each article element using a pre-trained dense vector model, with each dimension containing non-zero real values; the preset weight refers to a weighting coefficient pre-assigned according to the importance of each article element in representing the article's theme. In the embodiments of this invention, the title corresponds to 0.4, the abstract to 0.35, and the body text to 0.25; the dense vector refers to a unified vector representation obtained by weighting and summing the dense embedding vectors corresponding to each article element according to the preset weights and then normalizing them using the L2 norm; the initial sparse vector refers to a high-dimensional sparse representation containing term and weight key-value pairs obtained by using a sparse vector model to encode term activation for the article elements, which may contain illegal values ​​or extremely small noise components.

[0057] Furthermore, a distributed semantic encoding method based on a pre-trained language model can be used to semantically embed the article elements of the target financial news data to obtain dense embedding vectors. For example, a pre-trained dense vector model can be called to perform forward inference on the title text, summary text, and body text respectively, converting each text segment into a fixed-dimensional 1024-dimensional floating-point vector to obtain the corresponding dense embedding vector. Alternatively, a term activation encoding method based on a neural sparse retrieval model can be used to perform term activation encoding on the article elements of the target financial news data to obtain initial sparse vectors. For example, a pre-trained sparse vector model can be called to perform semantic embedding on the title, summary, and body text. The encoding process outputs a sparse representation containing each term in the vocabulary and its corresponding importance score, where only terms with scores greater than zero are retained as non-zero dimensions, resulting in an initial sparse vector. A dual-path filtering method based on outlier removal and threshold pruning can be used to perform quality filtering on the dimensional components of the initial sparse vector to obtain the final sparse vector. The specific steps are as follows: First, traverse all non-zero dimensional components in the initial sparse vector and directly remove components whose weight values ​​are non-numeric null values ​​or illegal values; second, set the smallest components whose absolute weight value is less than ten to the power of -15 to zero; finally, retain the remaining valid non-zero components to obtain the filtered sparse vector.

[0058] Optionally, as an optional embodiment of the present invention, the step of weighting and synthesizing the dense embedding vectors of each dimension based on preset weights to obtain a dense vector includes:

[0059] in, Represents a dense vector. This represents the dimensionally dense embedding vector corresponding to the i-th article element. Let represent the weight coefficient of the i-th article element, and n represent the number of article elements. Let represent the L2 norm, i represent the index of the article element, and n represent the total number of article elements.

[0060] This invention integrates deep semantic representation and precise keyword matching capabilities into a unified retrieval input by performing vector fusion on the dense vector and the sparse vector. This facilitates downstream search engines to simultaneously utilize both sources of information for mixed retrieval in the same query request, improving the recall and accuracy of cross-source semantic deduplication. The dual-path vector representation data refers to a unified data structure containing dual-path representation fields, obtained by encapsulating the dense vector field and the sparse vector field together. This structure is used by downstream search engines to simultaneously perform dense nearest neighbor search and sparse term matching.

[0061] Specifically, the vector fusion of the dense vector and the sparse vector to obtain dual-path vector representation data includes: The dense vector is serialized according to a floating-point array format of a preset dimension to obtain a dense vector field; The non-zero dimension components in the sparse vector are extracted into a set of key-value pairs of term identifiers and weight values ​​to obtain the sparse vector field. The dense vector field and the sparse vector field are encapsulated into a unified data structure containing dual-path representation fields to obtain dual-path vector representation data.

[0062] The dense vector field refers to a continuous numerical array formed by arranging 1024-dimensional dense floating-point vectors in a fixed order, which is used to store the dense vector field of the search engine; the sparse vector field refers to a key-value mapping table formed by extracting the term numbers and weight values ​​corresponding to all non-zero weight components in the sparse vector, which is used to store the sparse vector field of the search engine.

[0063] Furthermore, a dimension-flattening-based array serialization method can be used to serialize the dense vector according to a floating-point array format of a preset dimension to obtain the dense vector field. For example, the real values ​​of each dimension in the dense vector can be read sequentially and written into a floating-point array of length 1024 in order from the first dimension to the last dimension to obtain the dense vector field. Alternatively, a key-value extraction method based on non-zero item traversal can be used to extract the non-zero dimension components in the sparse vector into a set of key-value pairs of term identifiers and weight values ​​to obtain the sparse vector field. For example, all dimensions of the sparse vector can be scanned one by one, and when the absolute value of the weight of a certain dimension is not zero... When the index number of the dimension is used as the term identifier and the weight value of the dimension is used as the corresponding value, it is written into a key-value mapping set to obtain a sparse vector field. The dense vector field and the sparse vector field can be encapsulated into a unified data structure containing dual-path representation fields using a document assembly method based on dual-field mounting to obtain dual-path vector representation data. The specific steps are as follows: First, create a data container with fixed key names; second, assign the dense vector field to the dense vector key and the sparse vector field to the sparse vector key; finally, serialize the data container into a structured document format to obtain dual-path vector representation data.

[0064] The semantic deduplication filtering module 105 is used to perform semantic similarity retrieval in the financial news data based on the dual-path vector representation data, obtain a semantic similarity score, and perform deduplication filtering on the target financial news data based on the semantic similarity score to obtain a cleaned financial news dataset.

[0065] This invention, by performing semantic similarity retrieval on financial news data based on the dual-path vector representation data, can overcome the limitations of perfect matching of Uniform Resource Locators (URLs) and title texts. It can accurately identify semantically repetitive content describing the same financial event from news published by different sources, thereby expanding the deduplication coverage and reducing the false positive rate. The semantic similarity score refers to a scalar value mapped to the zero-to-one interval after normalization of the comprehensive similarity score. It is used to directly compare with a preset threshold to determine the final evaluation criterion for triggering the deduplication operation.

[0066] Specifically, based on the dual-path vector representation data, semantic similarity retrieval is performed in the financial news data to obtain a semantic similarity score, including: Extract the current dual-path vector and the historical dual-path vector from the dual-path vector representation data and the financial news data, respectively; Calculate the overall similarity between the current dual-path vector and the historical dual-path vector; The comprehensive similarity is normalized to obtain the semantic similarity score.

[0067] Wherein, the current dual-path vector and the historical dual-path vector are the vector field sets of the dual-path vector representation data and the financial news data, respectively. The comprehensive similarity refers to the comprehensive evaluation score obtained by merging the dense nearest neighbor search ranking and the sparse term matching ranking using the inverse ranking fusion algorithm. The higher the score, the greater the semantic repetition between the current article and the corresponding historical document.

[0068] Furthermore, a method based on direct reading using fixed key names can be used to extract the current and historical dual-path vectors from the dual-path vector representation data and the financial news data, respectively. For example, a 1024-dimensional floating-point array can be read from the dense vector key of the dual-path vector representation data as the dense part of the current dual-path vector, and a set of non-zero-dimensional key-value pairs can be read from the sparse vector key as the sparse part of the current dual-path vector. At the same time, the dense and sparse vector fields stored in the historical document objects corresponding to the financial news data can be read as historical dual-path vectors. A linear normalization method based on extreme value interval mapping can be used to normalize the comprehensive similarity to obtain a semantic similarity score. For example, the maximum and minimum values ​​of the comprehensive similarity of all historical documents can be obtained, the comprehensive similarity of each document can be subtracted from the minimum value, and then divided by the difference between the maximum and minimum values. The result can be mapped to the interval between zero and one to obtain the final semantic similarity score.

[0069] Furthermore, as an optional embodiment of the present invention, the comprehensive similarity between the current dual-path vector and the historical dual-path vector can be calculated using the following formula, including:

[0070] in, Indicates the overall similarity. This represents the current dense vector in the current two-way vector. Let represent the historical dense vector in the historical dual-path vector, and λ represent the preset fusion weight of the dense path. This represents the current sparse vector in the current dual-path vector. This represents the historical sparse vector in the historical bipath vector.

[0071] This invention performs deduplication filtering on the target financial news data based on the semantic similarity score. This automatically removes redundant articles that repeatedly report the same financial event in different sources, thereby reducing the amount of invalid documents stored by the search engine and improving the information diversity of the front-end search results. Furthermore, based on the semantic similarity score, the target financial news data is deduplicated using a threshold comparison and state update method to obtain a cleaned financial news dataset. For example, when the highest obtained semantic similarity score is 0.89, and the source type of the current article is a selected announcement, since 0.89 is greater than the preset threshold of 0.88 for this type of source, the corresponding historical document status field is updated to -99 to mark it as semantically duplicated and invalid. At the same time, the current article is written normally into the search engine index, resulting in a cleaned financial news dataset.

[0072] like Figure 3 The diagram shown is a flowchart illustrating a financial news data cleaning method according to an embodiment of the present invention. In this embodiment, the financial news data cleaning method includes: S1. Obtain raw data of multi-source heterogeneous financial news and read the stored financial news data. Perform normalization preprocessing on the raw data of multi-source heterogeneous financial news to obtain standardized financial news data. S2. Call the preset large language model intelligent agent to perform multi-topic semantic segmentation on the standardized financial news data to obtain a set of single-topic sub-articles, and perform semantic structuring processing on the set of single-topic sub-articles to obtain structured financial news data; S3. Match and verify the tags in the structured financial news data with the tag master data to obtain valid tags, and perform tag cleaning on the structured financial news data to obtain the target financial news data; S4. Perform dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors. Perform vector fusion on the dense vectors and the sparse vectors to obtain dual-path vector representation data. S5. Based on the dual-path vector representation data, perform semantic similarity retrieval in the financial news data to obtain a semantic similarity score. Based on the semantic similarity score, perform deduplication filtering on the target financial news data to obtain a cleaned financial news dataset.

[0073] In one embodiment, a computer device is provided, which may be a server or a client, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a financial news data cleaning system on the server or client side.

[0074] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0075] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0076] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

[0077] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0078] Finally, it should be noted that in the above embodiments, each embodiment can be combined with each other or independent. Deleting any one of them will not affect the technical implementation of other embodiments. The above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A financial news data cleaning system, characterized in that, The system includes: The data normalization and preprocessing module is used to acquire multi-source heterogeneous financial news raw data, read stored financial news data, and perform normalization and preprocessing on the multi-source heterogeneous financial news raw data to obtain standardized financial news data. The semantic segmentation and structuring module is used to call a preset large language model agent to perform multi-topic semantic segmentation on the standardized financial news data, obtain a set of single-topic sub-articles, and perform semantic structuring processing on the set of single-topic sub-articles to obtain structured financial news data. The tag verification and cleaning module is used to match and verify the tags in the structured financial news data with the tag master data to obtain valid tags, so as to perform tag cleaning processing on the structured financial news data to obtain the target financial news data. The dual-path vector representation module is used to perform dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors, and to perform vector fusion on the dense vectors and the sparse vectors to obtain dual-path vector representation data. The semantic deduplication filtering module is used to perform semantic similarity retrieval in the financial news data based on the dual-path vector representation data, obtain a semantic similarity score, and perform deduplication filtering on the target financial news data based on the semantic similarity score to obtain a cleaned financial news dataset.

2. The financial news data cleaning system as described in claim 1, characterized in that, The standardization preprocessing of the multi-source heterogeneous raw financial news data to obtain standardized financial news data includes: The multi-source heterogeneous financial news raw data is preprocessed using general rules to obtain general preprocessed data; Extract the source identifiers carried by the multi-source heterogeneous financial news raw data from the general preprocessed data; Based on the source identifier, the general preprocessed data is parsed with differentiated extended fields to obtain candidate label prompts and source structured fields; Based on the candidate tag suggestions and the source structured fields, the general preprocessed data is adapted to the source and normalized in format to obtain standardized financial news data.

3. The financial news data cleaning system as described in claim 2, characterized in that, Based on the source identifier, the general preprocessed data is parsed using differentiated extended fields to obtain candidate label hints and source structured fields, including: When the source identifier matches the selected source of the announcement, the stock code array is parsed from the extended field, verified with the stock master data and used as a candidate tag suggestion, and the influence, summary and announcement type fields are extracted as source structure fields; When the source identifier matches the research report source, the research report page number and tag list are parsed from the extended field, and the stock tag is suggested as a candidate tag after verification by the stock master data. When the source identifier matches the source of the surge interpretation, the subject name, subject driving logic and the number of stocks rising in the subject are parsed from the extended fields. The subject list is used as a candidate tag prompt and the subject details are used as the source structure field. When the source identifier matches a trending news source, the topic, stock, sorting index, and tag time are parsed from the extended fields, and the stored data can be queried by URL to determine whether there are duplicates. When the source identifier matches the chief opinion source of an organization, the tag list is parsed from the extended field, the source type identifier is set, and the knowledge planet jump address is fixed as the source structure field; When the source identifier does not match any configured dedicated processing logic, the word filtering service is invoked to extract stock entities mentioned in the article text based on financial dictionaries and regular expression rules. The extraction results are then verified against the stock master data and used as candidate tag suggestions.

4. The financial news data cleaning system as described in claim 1, characterized in that, The process involves invoking a pre-defined large language model agent to perform multi-topic semantic segmentation on the standardized financial news data, resulting in a single-topic sub-article set, including: Standardized financial news data is segmented into candidate semantic units at the paragraph and sentence group granularity, and topic segmentation prompt words are injected into the large language model agent. The agent labels the topic category to which each semantic unit belongs and the topic switching points between adjacent units. Based on the topic switching point, the candidate semantic units are aggregated into several candidate topic clusters, and the entity co-occurrence density and financial element coverage of each candidate topic cluster are statistically analyzed. If the co-occurrence density of entities within the same candidate topic cluster is lower than the preset agglomeration threshold, then backtrack and split the cluster, and merge the low-agglomeration fragments into the adjacent topic clusters; If the similarity of financial element coverage of adjacent candidate topic clusters is higher than the preset separation threshold, they will be merged into the same single topic sub-article. The candidate topic clusters, after being refined through cohesion and separation, are completed into independent single-topic sub-articles, resulting in a set of single-topic sub-articles.

5. The financial news data cleaning system as described in claim 1, characterized in that, The semantic structuring process performed on the single-topic sub-article set yields structured financial news data, including: Extract the title, body text, abstract, and impact fields from the single-topic sub-article collection; The summary field and the influence field are normalized to obtain a content summary and sentiment polarity identifier; The extracted title, body text, content summary, and sentiment polarity identifier are encapsulated according to a preset field structure to obtain structured financial news data.

6. The financial news data cleaning system as described in claim 1, characterized in that, The process of matching and validating the tags in the structured financial news data with the tag master data to obtain valid tags includes: Candidate tags in the structured financial news data are sorted by tag type to obtain a set of branched tags; Extract the tag codes from the branch tag set and query and match them with the stock index and theme index pre-stored in the tag master data to obtain preliminary matching tags; Metadata appending and invalid item removal are performed on the initially matched tags to obtain valid tags.

7. The financial news data cleaning system as described in claim 1, characterized in that, The step of performing tag cleaning on the structured financial news data based on the effective tags to obtain the target financial news data includes: Perform a deduplication operation on the valid tags according to the tag code to obtain a deduplicated tag set; The sub-articles with empty duplicate tag sets are removed from the structured financial news data to obtain the retained sub-article set; The retained sub-article set and the corresponding deduplication tag set are assembled into a structured document with fixed key names to obtain the target financial news data.

8. The financial news data cleaning system as described in claim 1, characterized in that, The process of performing dense vector encoding and sparse vector encoding on the article elements of the target financial news data to obtain dense vectors and sparse vectors includes: Semantic embedding is performed on each article element of the target financial news data to obtain a dense embedding vector; Based on preset weights, the dense embedding vectors of each dimension are weighted and synthesized to obtain a dense vector; The article elements of the target financial news data are subjected to term activation encoding to obtain an initial sparse vector; The dimensional components in the initial sparse vector are subjected to quality filtering to obtain a sparse vector.

9. The financial news data cleaning system as described in claim 1, characterized in that, The process of fusing the dense vector and the sparse vector to obtain dual-path vector representation data includes: The dense vector is serialized according to a floating-point array format of a preset dimension to obtain a dense vector field; The non-zero dimension components in the sparse vector are extracted into a set of key-value pairs of term identifiers and weight values ​​to obtain the sparse vector field. The dense vector field and the sparse vector field are encapsulated into a unified data structure containing dual-path representation fields to obtain dual-path vector representation data.

10. A method for cleaning financial news data, characterized in that, The method includes: Acquire raw financial news data from multiple sources and read stored financial news data. Perform normalization preprocessing on the raw financial news data from multiple sources and obtain standardized financial news data. The standardized financial news data is subjected to multi-topic semantic segmentation by a pre-defined large language model intelligent agent to obtain a set of single-topic sub-articles, and the single-topic sub-articles are subjected to semantic structuring processing to obtain structured financial news data. The tags in the structured financial news data are matched and verified with the tag master data to obtain valid tags, and then the structured financial news data is cleaned to obtain the target financial news data. Dense vector encoding and sparse vector encoding are performed on the article elements of the target financial news data to obtain dense vectors and sparse vectors. The dense vectors and sparse vectors are then fused to obtain dual-path vector representation data. Based on the dual-path vector representation data, semantic similarity retrieval is performed on the financial news data to obtain a semantic similarity score. Based on the semantic similarity score, the target financial news data is subjected to deduplication filtering to obtain a cleaned financial news dataset.

Images