Preprocessing method and system for packaged financial data based on rag processing

Abstract

The application discloses a financial data preprocessing method and system based on RAG processing packaging, and relates to the technical field of data processing. The method specifically comprises the following steps: receiving a source document for multi-modal analysis preprocessing, and outputting initial processing data; extracting document features to determine document complexity according to the initial processing data; performing dynamic adjustment of the slicing parameters; cutting the source document to output an initial text slicing sequence; marking references according to the initial text slicing sequence; outputting traceability data with quantitative confidence based on the text slicing sequence with reference marks; performing hierarchical enhancement and outputting enhanced text slices; checking, correcting and iteratively optimizing; and outputting standardized packaging and constructing a graph. The technical problems of semantic association fragmentation and information integrity damage caused by mechanical text slicing strategies when the existing technology based on retrieval enhancement generation (RAG) system processes logical rigorous documents such as finance are solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a financial data preprocessing method and system based on RAG processing packaging. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, natural language processing technology centered on large language models (LLMs) has made breakthrough progress, especially in processing massive and unstructured professional field documents, such as in the financial, legal, scientific research and many other knowledge-intensive industries, showing great application potential. To overcome the inherent knowledge update lag, factual errors and inability to process private domain knowledge of large language models, retrieval augmented generation (RAG) technology has emerged and quickly become a mainstream technology paradigm to improve model performance and reliability. This paradigm combines external knowledge bases with the generation capabilities of large language models, that is, dynamically retrieves relevant information from external knowledge bases and provides it as context to large language models, thereby significantly enhancing the factual accuracy, timeliness and domain expertise of model-generated answers. Especially in the financial field, RAG systems are widely used in intelligent investment research, compliance risk control, financial report analysis and other scenarios, and their processing capabilities for massive and unstructured financial documents directly affect the quality and efficiency of decision support.

[0003] The existing typical RAG technology process is usually divided into two core stages of knowledge base construction and query response. In the knowledge base construction stage, the system first receives external documents such as research reports, legal provisions or company annual reports, and uses a specific segmentation strategy to divide them into a series of text segments or "chunks". The original intention of this step is to convert long text into standardized modules with finer granularity that are suitable for model processing, to facilitate subsequent vectorization and retrieval. Specifically, the commonly used segmentation methods generally include mechanical division based on fixed character number, sentence or paragraph boundaries. After completing the segmentation, the system converts the above independent text segments into high-dimensional semantic vectors through an embedding model, and stores these vectors together with their corresponding original text segments in a special vector database for indexing to build a set of semantic knowledge space that can be efficiently retrieved.

[0004] However, as RAG technology is deeply applied in professional fields such as finance, which has extremely high requirements for logical rigor, analysis accuracy and information integrity, the inherent principle limitations of the above preprocessing paradigm centered on mechanical segmentation gradually emerge. The reason lies in the essential characteristics of financial documents (such as prospectuses, contract terms, etc.), which are highly related and structured internally. The text content forms a set of rigorous semantic networks through a large number of internal references, context references, logical progression and summary relationships.

[0005] The execution standard of the existing mechanical segmentation strategy design usually only focuses on the segmentation boundary of the text, the length and number of sentences or paragraph marks, ignores the deep logical association and semantic integrity of the document content, and the above processing method may cut off the inherent semantic chain of the original text, resulting in serious context fragmentation. For example, when the starting part of a group of segments appears "as mentioned above", "based on the above analysis" or "according to the foregoing provisions" and other expressions with explicit back-reference properties, the segment as a group of independent vectorization modules cannot accurately reflect the content, arguments or background of the preceding text on which it depends, and thus cannot fully and accurately carry the quoted context information, making it difficult to be effectively recalled in the subsequent retrieval stage, which directly leads to a serious decline in the quality of the final prompt delivered to the large language model, and may cause the model to generate misleading and even completely opposite results to the original intention, greatly damaging the reliability and practical value of the entire system. SUMMARY

[0006] Therefore, the present application provides a financial data preprocessing method and system based on RAG processing packaging to solve the technical problem that the mechanical text segmentation strategy may cause semantic association fragmentation and information integrity damage when the existing retrieval augmentation generation (RAG) system processes logic rigorous documents such as finance. It can perform automatic and multi-stage deep processing on the text segments after preliminary segmentation to accurately identify semantic associations, determine the source of reference relationships, adaptively enhance context information and perform closed-loop verification of logical consistency before document vectorization index construction, significantly improving the logical integrity, analysis accuracy and information reliability of the RAG system in the financial professional field.

[0007] In order to achieve the above purpose, the present application provides the following technical scheme:

[0008] A financial data preprocessing method based on RAG processing packaging, comprising the following steps:

[0009] Receiving a source document and performing multi-modal analysis preprocessing on the source document to output initial processing data;

[0010] According to the initial processing data, further extract document features to determine document complexity;

[0011] Based on the document complexity, the shard parameters are dynamically adjusted;

[0012] The source document is cut according to the dynamically adjusted shard parameters, and an initial text shard sequence is outputted;

[0013] According to the initial text shard sequence, explicit references, implicit references and cross-language references are marked;

[0014] Based on the text shard sequence marked with references, further output the provenance data with quantified confidence;

[0015] According to the provenance data with quantified confidence, perform hierarchical enhancement, and output the enhanced text shard;

[0016] Further, the enhanced text shard is checked, corrected and iteratively optimized;

[0017] According to the text shard and optimization data that pass the verification, output standardized encapsulation and build a graph.

[0018] On the basis of the above technical solutions, the present application is further described as follows:

[0019] As a further scheme of the present application,

[0020] The source document is received, and multi-modal analysis preprocessing is performed on the source document, and initial processing data is outputted, specifically including:

[0021] Receiving unstructured or semi-structured source documents;

[0022] The source document format includes PDF, DOCX, TXT, LaTeX;

[0023] Extracting structured information of the source document and converting multi-modal elements:

[0024] Finally, the initial processing data containing pure text, virtual text segments and source document structure information is outputted.

[0025] As a further scheme of the present application,

[0026] The document complexity is determined by further extracting document features according to the initial processing data, specifically including:

[0027] According to the received source document initial processing data, the average length of sentences of the source document, the proportion density of reference expressions, the proportion density of professional terms, the proportion of multi-modal elements and the language type are analyzed, and finally the document complexity score S is calculated and generated, and the specific calculation formula is as follows:

[0028] S = 0.3 x L n + 0.2 x Rr+ 0.2 x M + 0.15 x T + 0.15 x Lv

[0029] In the formula, L n is the average sentence length normalized value, which is calculated by the average semantic token number of a single sentence / the maximum threshold value of semantic tokens of a single sentence, so as to standardize the comparison of different sentence lengths and convert the absolute length value into a relative proportion value;

[0030] Rr is the reference expression proportion density, that is, the sum of the number of implicit references and explicit references in the source document, which is calculated by the total number of reference expressions in a specific number of words of the source document, for example, when the specific number of words of the source document is 1000, the total number of reference expressions in every 1000 words;

[0031] M is the multi-modal element proportion, which is calculated by (the sum of the number of pictures + the number of formulas + the number of tables) / the total number of paragraphs in the source document, wherein the number of pictures, formulas and tables in the source document and the total number of paragraphs are obtained through the initial processing data of the source document;

[0032] T is the professional term proportion density, which is calculated by the total number of professional terms / the total number of semantic tokens (tokens) of the source document;

[0033] Lv is the language type coefficient, wherein a single language document = 0.5 and a multi-language document = 1.0;

[0034] The multi-language type processing increases the complexity, and the weight is 0.15;

[0035] According to the document complexity score S generated by the final calculation, the document is divided, specifically:

[0036] When S is greater than or equal to 0.6, the document is classified as a high complexity score document;

[0037] When S is less than 0.6, the document is classified as a low complexity score document.

[0038] As a further scheme of the present application,

[0039] The document complexity based slicing parameter dynamic adjustment combines the dynamically adjusted slicing parameter to split the source document, and outputs an initial text slicing sequence, specifically including:

[0040] According to the high complexity score document or the low complexity score document classified, the initial target slicing processing parameter is dynamically adjusted: the target slicing length, the overlapping length between adjacent slices and the implicit reference recall K value;

[0041] For high complexity scoring documents, the target fragment length range is adjusted to 800-1024 character tokens, the overlap length is adjusted to 128 character tokens, and the implicit reference recall K value is adjusted to 5.

[0042] For low complexity scoring documents, the target fragment length range is adjusted to 300-400 character tokens, the overlap length is adjusted to 32 character tokens, and the implicit reference recall K value is adjusted to 2.

[0043] Then, a recursive character text segmentation algorithm is introduced, the target fragment length and the overlap length between adjacent fragments of the recursive character text segmentation algorithm are dynamically determined according to the document features, and when the recursive character text segmentation algorithm detects that the segmentation position may damage the context continuity, a recursive mechanism is triggered to backtrack and adjust the segmentation point, and the segmentation position is moved forward or backward to a complete semantic unit until the recursive character text segmentation algorithm detects that the text within each group of fragments is logically coherent.

[0044] The extracted pure text information and the text associated after the virtual text segment are cut into initial text fragments with dynamic overlap areas, and an initial text fragment sequence with a unique identifier, original document position metadata and multi-modal association mark is output; the unique identifier is the exclusive identification of each group of fragments, which is used for subsequent fragment association retrieval; the original document position metadata records the original position of each group of fragments in the source document for tracing; and the multi-modal association mark is the virtual text about the multi-modal elements contained in the fragment, which marks the association relationship with the original multi-modal elements.

[0045] As a further scheme of the present application, the following steps are further included:

[0046] According to the extracted document features, the corresponding multi-language combined expressions in the source document are obtained, and the multi-language combined expressions are further converted into a unified processing form, and then the vector conversion is performed to ensure that the semantics of different languages can be accurately compared in similarity, and finally the cross-language reference relationship is determined, which specifically includes:

[0047] According to the extracted document features, the corresponding multi-language combined expressions in the source document are obtained, and the text in different languages in the source document is converted into a vector in a unified semantic space, and then cross-language semantic similarity comparison is performed based on the vector, that is, after the text in different languages is converted into a vector in a unified semantic space, the closer the meanings between the two units of the reference tracing are, the more similar the arrangement rules of the vectors representing the two units are, so that the similarity of the two groups of vectors can be compared, and finally the cross-language reference relationship can be accurately obtained.

[0048] As a further scheme of the present application,

[0049] The explicit reference, implicit reference and cross-language reference corresponding to the initial text fragment sequence are specifically included.

[0050] According to the initial text fragment sequence, the identification scanning procedure is performed, and more specifically includes:

[0051] The explicit reference, that is, the expression in the initial text fragment that explicitly points to the target part, is identified. After identifying the explicit reference, the system uses a set of predefined XML style tags, that is, <REF_EXP> and < / REF_EXP>, as the starting tag and the ending tag to accurately wrap the explicit reference expression, and adds a set of target attributes in the starting tag as anchor points to store the recorded target position.

[0052] The implicit reference, that is, the expression that does not contain explicit pointing but has a backtracking reference relationship, is identified. After identifying the implicit reference, the system uses another set of tags, that is, <REF_IMP> and < / REF_IMP>, as the starting tag and the ending tag to accurately wrap the implicit reference expression, and assigns a globally unique temporary reference identifier to the implicit reference expression, which is stored in the id attribute to record the implicit reference expression by means of the unique identifier of the id attribute.

[0053] The cross-language reference, that is, the reference relationship between different language texts, is identified. After identifying the cross-language reference, the <REF_MLlang="en"target="Section2.1"> tag is additionally used to wrap the reference expression.

[0054] After the above processing, the text fragment anchored by the reference tag is finally output.

[0055] As a further scheme of the present application,

[0056] The text fragment sequence with reference tags is further output with quantified confidence traceability data, specifically including:

[0057] The text fragment anchored by the reference tag is received, and the source content corresponding to each group of reference pointing is further determined according to the reference tag and information. The traceability result is scored and compared according to the source content.

[0058] When a new document fragment is added, the vector index is updated in real time.

[0059] When the text fragment with the <REF_EXP> tag, that is, the explicit reference, is received, the structured target positioning engine is activated. It uses the document structure metadata saved when the initial text fragment is saved, and directly determines the initial fragment ID corresponding to the target position pointed by the target attribute through accurate string matching and logical rules. Since the explicit reference has explicit pointing, the traceability result has high accuracy.

[0060] When receiving a text segment with a <REF_IMP> tag (implicit reference) or a <REF_ML> tag (cross-language reference), the vectorized semantic retrieval engine is activated, and the workflow is as follows:

[0061] First, all initial text segments except the current text segment are converted into high-dimensional vectors by a set of text embedding models, which output 1024-dimensional vectors based on the transformer architecture and can accurately capture semantics after training on large-scale general, domain-specific, and multilingual parallel corpora.

[0062] Second, the text embedding model is used to convert the sentence with the reference tag in the current text segment into a query vector.

[0063] Finally, the cosine similarity between the query vector and all high-dimensional vectors is calculated based on the inverted file and product quantization approximate nearest neighbor search index structure, and the top K candidate segments with the most similar semantics are determined. The value of K is dynamically determined by the feature extraction adjustment module 20 according to the document complexity.

[0064] If it is a cross-language reference, the text embedding model is replaced by the mUSE model.

[0065] Further use of the confidence calculation module to score the confidence of the traceability results.

[0066] If the explicit reference is determined by the structured target positioning engine, the score is > 90.

[0067] If the implicit reference candidate and the cross-language reference candidate are determined by the vectorized semantic retrieval engine, the confidence score is calculated by a nonlinear mapping function:

[0068] This function maps the cosine similarity value in the range of 0 to 1 to the interval of 40 to 90, and the formula is only applicable when the cosine similarity value is greater than 0.7, otherwise the score is 0. The specific formula is as follows:

[0069] Score = 40 + 50 * ((Similarity - 0.7) / (1.0 - 0.7))

[0070] In the formula, Similarity is the cosine similarity value, and Similarity ≤ 0.7, then the score is 0.

[0071] In addition, the candidate segment with a cosine similarity value between 0.6 and 0.7 is marked as <FLAGtype="potential_relevant" / >, i.e., a potential relevant mark.

[0072] Finally output the data packet, which contains the current text segment ID, reference type, reference content, candidate source text segment ID list, corresponding confidence score list and potential related mark.

[0073] As a further scheme of the present application,

[0074] The hierarchical enhancement is performed according to the traceability data with quantified confidence, and the enhanced text segment is output, specifically including:

[0075] The traceability data with quantified confidence is received, and automatic content completion is performed for the current text segment based on the confidence score, and the more specific hierarchical processing process is as follows:

[0076] When the highest confidence score of a certain reference is greater than or equal to a preset first threshold (such as 80 points), it is determined that the reliability of the candidate source text segment meets the standard, and at this time the generative large language model reads the full text of the candidate source text segment with the highest confidence to generate a core content summary;

[0077] When the synchronization is a cross-language reference, the core content summary in the target language is generated synchronously;

[0078] The above core content summary is inserted into the original text segment and corresponds to the existing reference mark;

[0079] When the highest confidence score is less than 80 points but greater than or equal to a preset second threshold (such as 60 points), it is proved that the candidate source text segment has certain reliability, but there is uncertainty, at this time the continuous three sentences containing the most relevant keywords to the reference expression are extracted from the candidate source text segment with the highest confidence to form a original text segment; Then insert the original text segment behind the existing reference mark and introduce the "uncertain="true"" mark to clearly indicate that the supplemented content may have deviations for subsequent modules;

[0080] When the highest confidence score is less than 60 points but greater than or equal to the score corresponding to 0.6, it means that the candidate source text segment is potentially related but has low reliability, so the mark is inserted to correspond to the prompt for the subsequent module;

[0081] When the highest confidence score is less than 60 points, it means that reliable related content cannot be found for the time being, so the artificial review mark is inserted according to the existing reference mark.

[0082] As a further scheme of the present application,

[0083] The further verification and correction and iterative optimization are performed for the enhanced text segment, specifically including:

[0084] When it receives the content-enhanced text fragments, it inputs the original part and the enhanced part in the text fragments as a hypothetical group into the logical consistency evaluation model, then compares the original content and the enhanced content and outputs a logical consistency score between 0 and 1;

[0085] The closer the logical consistency score is to 1, the more matched the original content and the enhanced content are in logic; the lower the logical consistency score is, the greater the possibility of logical contradiction or irrelevance is;

[0086] A domain knowledge rule engine with a built-in extensible hard rule library defined in YAML format for a specific field is constructed, and the domain knowledge rule engine supports customizing rules through a visual interface, and the rules include name, trigger condition, verification logic and error prompt;

[0087] When verifying the text fragments, the domain knowledge rule engine checks against the rule library;

[0088] According to the score given by the logical consistency evaluation model and the checking result of the domain knowledge rule engine, the iterative correction process is further activated, specifically:

[0089] When the logical consistency score is lower than 0.5, or any domain rule is violated, the iterative correction operation is started, that is, the confidence of the hypothetical group formed by the original content and the enhanced content in the text fragment that causes the verification failure is reduced by 20%, then the updated data is returned, and the reference trace and adaptive enhancement process are triggered again for the corresponding hypothetical group, while the previous errors are excluded. In this way, the iterative correction is completed for each group of text fragments;

[0090] The upper limit of the number of iterations is dynamically determined according to the complexity of the document;

[0091] Finally, when the upper limit of the number of iterations is reached and the logical consistency and rule verification are still not passed, or the confidence of all candidate sources is lower than 60 points, the text fragment is marked and pushed to the artificial processing platform;

[0092] According to the verified text fragments and optimized data, a standardized encapsulation is output and a graph is constructed, specifically including:

[0093] Firstly, a marking and cleaning process is performed;

[0094] Specifically, all temporary reference marks and internal marks generated during the processing process are removed, and only the enhanced content marks with actual functions and processing status and risk prompt marks are retained;

[0095] Secondly, a standardized encapsulation process is performed;

[0096] Specifically, each set of processed text fragments is encapsulated into a structured JSON object, which is an information carrier configured with several clear fields as follows:

[0097] chunk_id is the unique identifier of the text fragment;

[0098] original_text records the original text content of the text fragment;

[0099] augmented_content stores augmented information, which contains augmented types such as abstracts or original text fragments, source text fragment IDs, and specific augmented content;

[0100] final_confidence is the final confidence score after multiple verifications and iterative corrections;

[0101] flags record special flags present;

[0102] multimodal_info is associated with multimodal element information;

[0103] Finally, a global fragment association graph is constructed;

[0104] When performing the standardization encapsulation process, a global fragment association graph in the form of a directed graph is simultaneously constructed;

[0105] Each node in the global fragment association graph corresponds to a text fragment, and each directed edge represents a verified reference relationship, i.e., from the referenced text fragment to the referenced text fragment, and the weight of the edge is set to the final confidence score.

[0106] A financial data preprocessing system for performing the financial data preprocessing method based on RAG processing encapsulation, the financial data preprocessing system comprising:

[0107] a document parsing and initial division module for receiving a source document and performing multimodal analysis and initial processing on the source document to output initial processing data;

[0108] a feature extraction and adjustment module for further extracting document features and determining document complexity based on the initial processing data, and dynamically adjusting the fragmentation parameters based on the document complexity;

[0109] The document parsing and initial division module is further configured to split the source document based on the dynamically adjusted fragmentation parameters to output an initial text fragment sequence;

[0110] The multi-language processing module is used for extracting document features, obtaining corresponding multi-language combined expressions in the source document, and further converting the multi-language combined expressions into a unified processing form, and then converting vectors to ensure that the semantics of different languages can be accurately compared in similarity, and finally determining the cross-language reference relationship.

[0111] The semantic association recognition anchoring module is used for marking explicit references, implicit references and cross-language references according to the initial text fragment sequence.

[0112] The reference tracing and confidence quantification module is used for further outputting tracing data with quantified confidence based on the text fragment sequence with reference marks.

[0113] The adaptive content enhancement module is used for performing hierarchical enhancement according to the tracing data with quantified confidence, and outputting enhanced text fragments.

[0114] The verification and correction module is used for verifying and iteratively optimizing the enhanced text fragments.

[0115] The encapsulation and graph construction module is used for outputting standardized encapsulation and constructing a graph according to the verified text fragments and optimization data.

[0116] The present application has the following beneficial effects:

[0117] The method and system can effectively improve the adaptability to multi-modal documents and multi-language scenarios through the constructed multi-stage cooperative operation module. Through deep analysis, intelligent tracing and dynamic enhancement processes, the module can automatically identify and repair the cross-fragment semantic association within the document before document vectorization indexing, intelligently complete the content and reconstruct the logic of the document fragments, thereby fundamentally solving the semantic fragmentation problem caused by mechanical fragmentation in the existing RAG process while retaining the advantages of fragmentation processing, and improving the accuracy, reliability and deep understanding ability of the RAG system in processing complex professional documents. BRIEF DESCRIPTION OF DRAWINGS

[0118] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. The structure, proportion, size, etc. shown in the present specification are only used to cooperate with the content disclosed in the specification, so that those skilled in the art can understand and read. Any modification of structure, change of proportion relationship or adjustment of size, which does not affect the effects and purposes that can be achieved by the present application, should still fall within the scope of the technical content disclosed by the present application.

[0119] Fig. 1 The overall flowchart of the financial data preprocessing method based on RAG processing encapsulation provided by the embodiments of the present application is shown.

[0120] Fig. 2 A functional architecture principle schematic diagram of a financial data preprocessing system based on RAG processing encapsulation is provided for an embodiment of the present application.

[0121] Fig. 3 An electronic device entity structure schematic diagram is provided for an embodiment of the present application.

[0122] In the drawings, the component list represented by each reference numeral is as follows:

[0123] The document analysis preliminary division module 10, the feature extraction adjustment module 20, the multi-language processing module 30, the semantic association recognition anchoring module 40, the reference tracing and confidence quantification module 50, the adaptive content enhancement module 60, the verification correction module 70, the man-machine collaborative optimization module 80, and the encapsulation and graph construction module 90. DETAILED DESCRIPTION

[0124] The embodiments of the present application are described below by specific embodiments, and those skilled in the art can easily understand other advantages and effects of the present application from the content disclosed in the present specification. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0125] The terms such as "upper", "lower", "left", "right", "middle" and the like cited in the present specification are only for the convenience of clear description, not to limit the scope of the present application, and the change or adjustment of the relative relationship is also considered as the implementable scope of the present application without substantial change of the technical content.

[0126] As shown in Figs. 1 to 3 The embodiments of the present application provide a financial data preprocessing method and system based on RAG processing encapsulation, to construct an automatic operation process of a plurality of specialized processing modules working together, to effectively solve the deep semantic association fracture problem caused by document physical fragmentation in the traditional retrieval augmentation generation (RAG) process, and to further introduce dynamic parameter adjustment, multi-modal analysis and multi-language adaptation collaborative optimization mechanism by simulating the deep cognitive understanding process to reconstruct and enhance the document fragments, so as to reduce the semantic misdirection caused by fragmentation from the root, and to ensure the logical integrity, accuracy and information reliability of the data module finally input to the large language model for vectorization indexing. The specific settings are as follows:

[0127] The embodiments of the present application provide a financial data preprocessing system based on RAG processing encapsulation, which is functionally divided at the software level and specifically includes:

[0128] The document parsing preliminary division module 10, the feature extraction adjustment module 20, the multi-language processing module 30, the semantic association recognition anchoring module 40, the reference tracing and confidence quantification module 50, the adaptive content enhancement module 60, the verification correction module 70, the man-machine collaborative optimization module 80 and the encapsulation and atlas construction module 90.

[0129] The document parsing preliminary division module 10 is configured to receive original, unstructured or semi-structured source documents, and the format of the source documents includes but is not limited to portable document format (PDF), Office open XML document format (DOCX), plain text format (TXT) and formula-containing academic document format (LaTeX).

[0130] Specifically, the document parsing preliminary division module 10 internally integrates at least one set of document structure parsers and at least one set of multi-modal analysis sub-modules; wherein the document structure parsers can identify and extract structured information such as pure text content, chapter titles, paragraphs, lists and tables of the document; the multi-modal analysis sub-modules further integrate an OCR recognition unit for PDF text extraction, a formula recognition unit for extracting formula semantic association text description based on Mathpix technology, and an image description generation unit for generating picture-associated text description based on the BLIP model, to convert non-text elements into processable virtual text segments through the multi-modal analysis sub-modules, and associate the virtual text segments to the corresponding text positions.

[0131] The feature extraction adjustment module 20 is connected to the document parsing preliminary division module 10 through data, and is used for further completing feature extraction and parameter dynamic adjustment for the received source document, specifically including:

[0132] Because different types of source documents differ greatly in structural complexity, semantic density and reference relationship, fixed parameters cannot balance processing effect and efficiency. Therefore, the source document type is pre-identified and judged;

[0133] By analyzing the average length of the sentences of the received source document, the reference expression proportion density (such as the density of “as mentioned above” and “based on the above analysis” reference expressions per thousand characters), the professional term proportion density, the multi-modal element proportion (such as the density of picture and formula expressions) and the language type (single language / multi-language), a document complexity score S is finally calculated and generated, and the specific calculation formula is as follows:

[0134] S = 0.3×L n + 0.2×Rᵣ+ 0.2×M + 0.15×T + 0.15×Lᵥ

[0135] In the formula, L nThe average sentence length normalization value is calculated as the average semantic token number of a single sentence / the maximum threshold value of semantic tokens of a single sentence, so as to standardize the comparison of different sentence lengths, and to convert the absolute length value into a relative proportion value, rather than using a fixed value calculation;

[0136] The higher the proportion of long sentences in the source document, the more complex the structure of the source document (such as an academic paper), and the weight is 0.3;

[0137] Rᵣ is the proportion density of citation expressions, that is, the sum of the number of implicit citations and explicit citations in the source document, and the calculation method is the total number of citation expressions in a specific number of words in the source document, such as 1000 words in the source document. The total number of citation expressions appears in every 1000 words;

[0138] The greater the proportion density of citation expressions, the longer the logical chain of the source document and the higher the complexity, and the weight is 0.2;

[0139] M is the proportion of multi-modal elements, and the calculation method is (the sum of the number of pictures + the number of formulas + the number of tables) / the total number of paragraphs in the source document, wherein the number of pictures, formulas and tables are identified and obtained by the multi-modal analysis submodule, and the total number of paragraphs in the source document is identified and obtained by the document structure parser;

[0140] The more multi-modal elements in the source document (such as formulas and tables in academic papers), the higher the processing complexity, and the weight is 0.2;

[0141] T is the proportion density of professional terms, and the calculation method is the total number of professional terms / the total number of semantic tokens (the smallest semantic unit when the model performs semantic understanding) in the source document;

[0142] The more professional terms in the source document (such as legal documents), the higher the processing difficulty, and the weight is 0.15;

[0143] Lᵥ is the language type coefficient, wherein single language document = 0.5, and multi-language document = 1.0;

[0144] The multi-language type processing increases the complexity, and the weight is 0.15.

[0145] According to the document complexity score S generated by the final calculation, the document is divided, specifically:

[0146] When S≥0.6, it is divided into a high complexity score document, such as a legal document, an academic paper, etc.;

[0147] When S<0.6, it is divided into a low complexity score document, such as a contract, a financial report, etc.

[0148] The initial target sharding processing parameters that need to be dynamically adjusted according to the document complexity score include:

[0149] Target fragment length: number of semantic tokens of a single text fragment;

[0150] Overlap length between adjacent fragments: number of semantic tokens of repeated content between adjacent fragments, used to maintain context continuity, such as transitional sentences between paragraphs;

[0151] Implicit reference recall K value: number of candidate source text fragments recalled when searching for implicit references.

[0152] Adjust the initial target fragment processing parameters based on the document complexity score, specifically:

[0153] For high complexity score documents, adjust the target fragment length range to 800-1024 token characters, the overlap length to 128 token characters, and the implicit reference recall K value to 5. This can reduce the fragmentation of long logical chains by using longer fragment lengths, avoid breaking single sentences into multiple fragments, maintain more context associations between adjacent fragments by configuring a larger overlap length, and cover more complex implicit references (such as cross-fragment references of "multiple chapter conclusions in the previous text", which cannot be directly determined by pointing identifiers and are therefore implicit references, and vice versa).

[0154] For low complexity score documents, adjust the target fragment length range to 300-400 token characters, the overlap length to 32 token characters, and the implicit reference recall K value to 2. This can avoid information redundancy by using shorter fragment lengths, reduce repeated content by configuring a smaller overlap length due to the simple logic that does not require a large amount of overlap to maintain continuity, and cover a small number of implicit references such as single reference "previous event" while improving processing efficiency.

[0155] After that, the document parsing initial division module 10 receives the initial target fragment processing parameters dynamically adjusted by the feature extraction adjustment module 20, and introduces a recursive character text segmentation algorithm. The target fragment length and overlap length between adjacent fragments of the recursive character text segmentation algorithm are dynamically determined by the feature extraction adjustment module 20 according to the document features, and when the algorithm detects that the division position may damage the context continuity, the recursive mechanism is triggered, the division point is adjusted back, and the division position is moved forward or backward to the complete semantic unit until the text logic within each group of fragments is detected to be coherent. At the same time, by setting the overlap area of adjacent fragments, it is further ensured that key information is preserved between each group of fragments, avoiding semantic fragmentation caused by division.

[0156] The pure text information extracted by the document parsing preliminary module 10 and the text associated after the virtual text segment are cut into initial text segments with dynamic overlapping areas, and a set of initial text segment sequences with unique identifiers (ChunkID), original document position metadata and multi-modal association marks are output, wherein the unique identifier is the exclusive identification of each group of segments, which is used for subsequent segment association retrieval; the original document position metadata is the original position of each group of segments in the source document to facilitate tracing; the multi-modal association mark is the virtual text about the multi-modal elements contained in the segment, which marks the association relationship with the original multi-modal elements (such as formula number) to ensure that the multi-modal information is not lost.

[0157] For multi-language type documents, the automatic synchronization triggers the adaptation process of the multi-language processing module 30, specifically:

[0158] Because multi-language documents such as Chinese-English mixed contracts and bilingual reports have cross-language references, such as "Chinese reference sentences" corresponding to "English referenced segments", or semantic alignment (identifying different languages with the same semantics) and other special requirements, if the single-language document processing scheme is still used, it may cause the cross-language reference association to be cut off, so the multi-language processing module 30 is introduced for multi-language documents;

[0159] The input end of the multi-language processing module 30 is connected to the feature extraction adjustment module 20 through data connection, to obtain corresponding multi-language combined expressions in the source document based on the feature extraction adjustment module 20, and further convert the multi-language combined expressions into a unified processing form, and then convert the vectors to ensure that the semantics of different languages can be accurately compared for similarity, and finally determine the cross-language reference relationship, specifically including:

[0160] The multi-language processing module 30 integrates a multi-language word segmentation model (XLM~RoBERTa) to uniformly segment the multi-language combined expressions, that is, to divide the multi-language text into uniform semantic units;

[0161] The multi-language processing module 30 also integrates an mUSE model to convert texts in different languages into vectors in a unified semantic space through the mUSE model in the reference tracing process, and then compare the cross-language semantic similarity based on the vectors, that is, after converting texts in different languages into vectors in a unified semantic space, the closer the meanings of the two units in the reference tracing are, the more similar the arrangement rules of the vector representations are, so that the similarity of the two sets of vector representations can be compared to finally accurately obtain the cross-language reference relationship;

[0162] The semantic association recognition anchoring module 40 has its input terminals respectively connected to the output terminals of the document parsing preliminary division module 10 and the multilingual processing module 30, and the semantic association recognition anchoring module 40 is internally provided with a sequence labeling model fine-tuned for a specific task. The bottom layer architecture of the sequence labeling model is a transformer network with an encoder-decoder structure, and a language adaptation layer is added for a multilingual scenario. The sequence labeling model is trained based on a professional field corpus, and the labeling information in the corpus accurately defines the boundaries and types of explicit references, implicit references, and cross-language references.

[0163] The semantic association recognition anchoring module 40 processes the initial text segment sequence from the document parsing preliminary division module 10, and performs a recognition scanning procedure for each group of initial text segment sequences, specifically including:

[0164] Explicit references are recognized, i.e., expressions in the initial text segment that explicitly point to target parts, such as “see Chapter 3, Section 2”, “as shown in equation (3.1) above”, etc. After recognizing the explicit reference, the system uses a set of predefined XML style tags, i.e., <REF_EXP> and < / REF_EXP>, as starting and ending tags to accurately wrap the explicit reference expression, and adds a set of target attributes to the starting tag as anchor points to store the recorded target location, for example, <REF_EXP target="Chapter 3, Section 2">.

[0165] Implicit references are recognized, i.e., expressions that do not contain explicit pointing but have backtracking reference relationships, such as “based on the above analysis”, “the method has the following defects”, etc. After recognizing the implicit reference, the system uses another set of tags, i.e., <REF_IMP> and < / REF_IMP>, as starting and ending tags to accurately wrap the implicit reference expression, and assigns a set of globally unique temporary reference identifiers to the implicit reference expression, which are stored in the id attribute, to record the implicit reference expression by means of the unique identifier of the id attribute, for example, <REF_IMP id="IMP_REF_0071">.

[0166] Cross-language references are recognized, i.e., reference relationships between different language texts, such as “as described in the English abstract”, “refer to the analysis in Section 2.1”, etc. After recognizing the cross-language reference, an additional <REF_ML lang="en" target="Section 2.1"> tag is used to wrap the reference expression, which not only clearly indicates the language type of the reference, but also records the reference target location.

[0167] After the above processing, the final output is a text segment with accurate reference marking, which enables the subsequent module to more accurately and clearly recognize the reference type and related information.

[0168] As a preferred embodiment, the input end of the multi-language processing module 30 is connected to the output end of the semantic association recognition anchoring module 40 by data, so that the sequence labeling model in the semantic association recognition anchoring module 40 can be called during the reference tracing process of the multi-language processing module 30, and then the training results of the professional field corpus are utilized by the sequence labeling model, so that the multi-language processing module 30 can more accurately recognize the cross-language reference features, and when there is a cross-language implicit reference, it is preferred to judge whether there is other reference possibility, so as to avoid missing cross-language association and further enhance the logical continuity of multi-language documents.

[0169] The reference tracing and confidence quantification module 50 is connected to the output end of the semantic association recognition anchoring module 40 for receiving the text segment anchored by the reference mark from the semantic association recognition anchoring module 40, and further determining the source content pointed by each group of reference marks according to the reference mark and information, and comparing the scores of the tracing results according to the source content to determine the reliability of the source content.

[0170] Specifically, the reference tracing and confidence quantification module 50 integrates a group of dynamic index sub-modules, two parallel groups of processing engines, and a group of confidence calculation modules; wherein the dynamic index sub-module is internally provided with an incremental update mechanism based on the dynamic index structure of FAISS, so that the vector index can be updated in real time when a new document segment is added, without the need for full index reconstruction when processing large-scale documents, which significantly improves the efficiency.

[0171] The two parallel groups of processing engines include a structured target positioning engine and a vectorized semantic retrieval engine.

[0172] When receiving a text segment with a <REF_EXP> mark (explicit reference), the structured target positioning engine is activated, which uses the document structure metadata saved when the initial text segment is saved, and directly determines the target position corresponding to the initial segment ID by precise string matching and logical rules. Since the explicit reference is clear, the tracing result is highly accurate.

[0173] When receiving a text segment with a <REF_IMP> mark (implicit reference) or a <REF_ML> mark (cross-language reference), the vectorized semantic retrieval engine is activated, and the workflow is as follows:

[0174] Firstly, all initial text fragments except the current text fragment are converted into high-dimensional vectors by a set of text embedding models, which output 1024-dimensional vectors based on the transformer architecture and can accurately capture semantics after being trained on large-scale general, domain-specific, and multilingual parallel corpora;

[0175] Secondly, the sentence with the reference mark of the current text fragment is converted into a query vector using the text embedding model;

[0176] Finally, the cosine similarity between the query vector and all high-dimensional vectors is calculated based on the inverted file and product quantization-based approximate nearest neighbor (ANN) index structure, and the top K candidate fragments with the most similar semantics are determined, with the value of K being dynamically determined by the feature extraction adjustment module 20 according to the document complexity;

[0177] If it is a cross-language reference, the text embedding model is replaced by the mUSE model.

[0178] Further, the confidence score calculation module is used to score the confidence of the traceability results for comparison;

[0179] If the explicit reference is determined by the structured target positioning engine, the score is > 90 points;

[0180] If the implicit reference candidate and the cross-language reference candidate are determined by the vectorized semantic retrieval engine, the confidence score is calculated by a nonlinear mapping function:

[0181] This function maps the cosine similarity value in the range of 0 to 1 to the interval of 40 to 90 points, and this formula is only applicable when the cosine similarity value is greater than 0.7, otherwise the score is 0; the specific formula is as follows:

[0182] Score = 40 + 50*((Similarity-0.7) / (1.0-0.7))

[0183] In the formula, Similarity is the cosine similarity value, and Similarity≤0.7, then the score is 0 points;

[0184] In addition, the candidate fragments with cosine similarity values between 0.6 and 0.7 are marked as <FLAGtype="potential_relevant" / >, i.e., potential relevant markers.

[0185] Finally, the reference traceability and confidence quantification module 50 outputs a data packet, which contains the current text fragment ID, the reference type, the reference content, the candidate source text fragment ID list, the corresponding confidence score list, and the potential relevant marker, to provide a basis for the subsequent content enhancement process.

[0186] The adaptive content enhancement module 60 is connected to the output end of the reference traceability and confidence quantification module 50, and has a built-in generative large language model (Qwen~7B~MoE) with 70 billion parameters, supports multi-language generation, and can complete text summary generation, key information extraction, and cross-language translation assistance under given context and constraint conditions after instruction fine-tuning.

[0187] The adaptive content enhancement module 60 receives the confidence score from the reference traceability and confidence quantification module 50, and automatically completes the content based on the confidence score for the current text segment, so that the semantics of the text segment is more complete. The specific hierarchical processing flow of the candidate source text segment is as follows:

[0188] When the highest confidence score of a certain reference is greater than or equal to a preset first threshold (such as 80 points), the reliability of the determined candidate source text segment meets the standard, and the adaptive content enhancement module 60 reads the full text of the candidate source text segment with the highest confidence based on the generative large language model to generate a core content summary. When it is a cross-language reference, a core content summary in the target language is generated synchronously;

[0189] Insert the above core content summary into the original text segment and mark the existing reference;

[0190] When the highest confidence score is less than 80 points but greater than or equal to a preset second threshold (such as 60 points), it proves that the candidate source text segment has certain reliability but has uncertainty. At this time, the three consecutive sentences containing the most relevant keywords to the reference expression are extracted from the candidate source text segment with the highest confidence to form a text fragment. Then, the text fragment is inserted after the existing reference mark in the format and a "uncertain="true"" mark is introduced to indicate that the supplement content may have bias for subsequent modules;

[0191] When the highest confidence score is less than 60 points but greater than or equal to the score corresponding to 0.6, it means that the candidate source text segment is potentially related but has low reliability. At this time, the mark is inserted to prompt the subsequent module accordingly;

[0192] When the highest confidence score is less than 60 points, it means that reliable related content cannot be found temporarily, so the review mark is inserted according to the existing reference mark to remind manual intervention.

[0193] Through the above hierarchical automatic processing, the adaptive content enhancement module 60 can supplement content or mark prompts according to different confidence situations, which not only ensures the effectiveness of the supplemented content, but also timely discovers human intervention, so that the semantics of the text fragments are more complete and reliable, and provide a sufficient foundation for subsequent processing.

[0194] The check correction module 70 is connected to the output end of the adaptive content enhancement module 60 through the input end data, which can form a quality check to effectively ensure the quality of the text fragments, so that the logic of the text fragments after adaptive content enhancement is more continuous and standardized, and the processing result can be continuously optimized through feedback loop.

[0195] Specifically, the check correction module 70 includes a logical consistency evaluation model, a domain knowledge rule engine, an iterative control module, and a man-machine cooperation interface submodule.

[0196] The logical consistency evaluation model is built-in with a classification model trained based on massive texts through contrastive learning; when it receives the text fragments after content enhancement from the adaptive content enhancement module 60, the logical consistency evaluation model inputs the original part and the enhanced part in the text fragments as a hypothetical group, then compares the original content and the enhanced content and outputs a logical consistency score between 0 and 1.

[0197] The closer the above logical consistency score is to 1, the more matched the original content and the enhanced content are in logic; the lower the logical consistency score is, the greater the possibility of logical contradiction or irrelevance is.

[0198] The domain knowledge rule engine is built-in with an extensible hard rule library defined in YAML format for specific fields, and supports customizing rules through a visual interface. The rules include name, trigger condition, check logic, and error prompt.

[0199] When checking the text fragments, the domain knowledge rule engine checks against the rule library.

[0200] Different fields have different professional rules. For example, in legal documents, the rule can be "If the reference expression is 'the provisions of the preceding paragraph', the physical location of the reference source fragment in the document must be immediately before the current fragment."

[0201] The iterative control module can further activate the iterative correction process according to the score given by the logical consistency evaluation model and the checking result of the domain knowledge rule engine, specifically:

[0202] When the logical consistency score is below 0.5, or any domain rule is violated, the iteration control module initiates a correction operation, i.e. reduces the confidence of the hypothetical group formed by the original content and the enhanced content in the text segment that causes the verification failure by 20%, and then returns the updated data to the reference tracing and confidence quantification module 50, and triggers the reference tracing and adaptive enhancement process again for the hypothetical group, while excluding the previous error correspondence, so as to complete the iteration correction for each group of text segments;

[0203] The upper limit of the number of iteration corrections is dynamically determined by the feature extraction adjustment module 20 according to the document complexity.

[0204] Finally, when the upper limit of the number of iteration corrections is reached and the logical consistency and rule check are still not passed, or the confidence of all candidate sources is below 60 points, the text segment is marked and pushed to the artificial processing platform through the human-computer cooperation interface submodule, which is used to receive the results of artificial correction, such as correct reference sources, supplementary context and other correction data, and can synchronize the above correction data to the human-computer cooperation optimization module 80 to provide basis for subsequent optimization of the system.

[0205] The human-computer cooperation optimization module 80, whose input end is connected to the human-computer cooperation interface submodule of the verification and correction module 70, can realize continuous optimization of the system based on artificial correction data, specifically:

[0206] An artificial correction corpus is constructed to store records of artificial marked correct reference relationships, context supplements and rule correction; the artificial correction corpus is periodically used to fine-tune the sequence labeling model of the semantic association recognition anchor module 40 (to optimize the accuracy of reference recognition), and the confidence calculation function of the reference tracing and confidence quantification module 50 (to adjust the scoring mapping relationship); in addition, special rules corrected by artificial can be automatically converted into new rules of the domain knowledge rule engine to realize dynamic updating of the rule library.

[0207] The packaging and graph construction module 90, whose input end is connected to the output end of the verification and correction module 70 and the human-computer cooperation optimization module 80, is used to receive the iteration correction data output by the verification and correction module 70 and / or the human-computer cooperation optimization module 80, and convert it into standardized data for subsequent use of the RAG system. It includes two processes: standardized packaging and global segment association graph construction, specifically as follows:

[0208] First, a marking and cleaning process is performed;

[0209] Specifically, all temporary reference marks (such as <REF_IMP> marks) and internal marks generated during processing are removed, only the enhanced content marks (such as marks for supplementary abstracts or original text fragments) with actual functions and processing status and risk prompt marks (such as marks indicating low reliability) are retained, so as to effectively avoid redundant mark interference on the downstream system on the basis of retaining key information.

[0210] Secondly, the standard encapsulation process is performed.

[0211] Specifically, the encapsulation and graph construction module 90 encapsulates each set of processed text fragments into a structured JSON object, and the JSON object is an information carrier, which is configured with a plurality of clear fields as follows:

[0212] chunk_id is used as a unique identifier of the text fragment;

[0213] original_text records the original text content of the text fragment;

[0214] augmented_content stores the augmented information, which includes the augmented type, the source text fragment ID, and the specific augmented content;

[0215] final_confidence is the final confidence score after multiple verification and iterative correction;

[0216] flags record the special marks present;

[0217] multimodal_info is associated with multimodal element information, such as the ID of the formula and the ID of the picture description; in this way, the scattered processing results are converted into standard data packets, so that the subsequent RAG system can quickly extract the required information.

[0218] Finally, the global fragment association graph is constructed.

[0219] During the standard encapsulation process, the encapsulation and graph construction module 90 synchronously constructs a directed graph form of a global fragment association graph, each node in the global fragment association graph corresponds to a text fragment, each directed edge represents a verified reference relationship, that is, the reference text fragment points to the referenced text fragment, and the weight of the edge is set to the final confidence score; the global fragment association graph is saved in the GraphML format, and by drawing the reference relationship between the text fragments into a clear graph, it is more convenient for the downstream system to directly use, and at the same time, the efficiency and accuracy of subsequent retrieval and generation can be significantly improved.

[0220] Fig. 3 The electronic device entity structure schematic diagram of the embodiment of the present application is as follows:Fig. 3 As shown, the electronic device includes a processor, a memory and an internal bus; wherein the processor, the memory complete the communication between each other through the internal bus;

[0221] The processor is configured to invoke program instructions in the memory to execute the methods provided by the above-mentioned method embodiments, for example, including: receiving a source document, and performing multi-modal analysis initial processing on the source document to output initial processing data; further extracting document features to determine document complexity according to the initial processing data; dynamically adjusting the slicing parameters based on the document complexity; combining the dynamically adjusted slicing parameters to split the source document to output an initial text slicing sequence; marking explicit references, implicit references and cross-language references according to the initial text slicing sequence; further outputting traceability data with quantified confidence based on the text slicing sequence with reference marking; performing hierarchical enhancement according to the traceability data with quantified confidence, and outputting enhanced text slices; further verifying, correcting and iteratively optimizing the enhanced text slices; outputting standardized encapsulation and constructing a graph according to the text slices that pass the verification and optimization data.

[0222] The embodiment provides a non-transitory computer-readable storage medium storing computer instructions, the computer instructions causing a computer to execute the methods provided by the above-mentioned method embodiments, for example, including: receiving a source document, and performing multi-modal analysis initial processing on the source document to output initial processing data; further extracting document features to determine document complexity according to the initial processing data; dynamically adjusting the slicing parameters based on the document complexity; combining the dynamically adjusted slicing parameters to split the source document to output an initial text slicing sequence; marking explicit references, implicit references and cross-language references according to the initial text slicing sequence; further outputting traceability data with quantified confidence based on the text slicing sequence with reference marking; performing hierarchical enhancement according to the traceability data with quantified confidence, and outputting enhanced text slices; further verifying, correcting and iteratively optimizing the enhanced text slices; outputting standardized encapsulation and constructing a graph according to the text slices that pass the verification and optimization data.

[0223] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware, the aforementioned program can be stored in a computer readable storage medium, the program executes to execute steps including the above-mentioned method embodiments; and the aforementioned storage medium includes ROM, RAM, magnetic disc or optical disc and various storage media that can store program codes.

[0224] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separate, and the components illustrated as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0225] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part that contributes to the prior art, can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to cause a computer device (which can be a server, or a network device, etc.) to execute the methods of the embodiments or some parts of the embodiments.

[0226] Although the present application has been fully described by way of examples with reference to the accompanying drawings, it is apparent that various modifications and changes can be made thereto without departing from the spirit of the application. Therefore, the present application should not be limited to the embodiments described herein, but should be given the widest scope in accordance with the principles and novel features disclosed herein.

Claims

1. A financial data preprocessing method based on RAG processing encapsulation, characterized in that, Includes the following steps: Receive the source document, perform multimodal parsing and initial processing on the source document, and output the initial processed data; Based on the initial processed data, further extract document features to determine document complexity; Dynamically adjust sharding parameters based on document complexity; The source document is segmented by dynamically adjusted segmentation parameters, and the initial text segmentation sequence is output. Explicit references, implicit references, and cross-language references are marked according to the initial text segment sequence; Based on the text fragment sequence with citation tags, further source data with quantified confidence is output; Perform hierarchical enhancement based on the source data with quantified confidence levels, and output the enhanced text fragments; Further verification, correction, and iterative optimization were performed on the enhanced text segmentation. Based on the validated text fragments and optimized data, output standardized encapsulation and construct a graph; The step of further extracting document features to determine document complexity based on the initial processed data specifically includes: Based on the initial processing data of the received source documents, the average sentence length, density of citations, density of technical terms, proportion of multimodal elements, and language type of the source documents are analyzed. Finally, the document complexity score S is calculated, and the specific calculation formula is as follows: S = 0.3×L n + 0.2×Rᵣ + 0.2×M + 0.15×T + 0.15×Lᵥ In the formula, L n This is a normalized value for the average sentence length, calculated as the average number of semantic characters per sentence / the maximum threshold of semantic characters per sentence. This allows for standardized comparisons of different sentence lengths and converts absolute length values ​​into relative proportion values. Rᵣ represents the citation density, which is the ratio of the sum of implicit and explicit citations in the source document. It is calculated as the total number of citations in a specific number of words in the source document. When the specific number of words in the source document is 1000, it is the total number of citations in every 1000 words. M represents the proportion of multimodal elements, calculated as (the sum of the number of images, formulas, and tables) / the total number of paragraphs in the source document. The number of images, formulas, and tables in the source document, as well as the total number of paragraphs, are obtained from the initial processing data of the source document. T represents the density of technical terms, which is calculated as the total number of technical terms / the total number of semantic characters in the source document. Lᵥ is the language type coefficient, where monolingual documents = 0.5 and multilingual documents = 1.0; Multi-language type handling increases complexity, with a weight of 0.15; Documents are divided according to the final calculated document complexity score S, specifically as follows: When S ≥ 0.6, it is classified as a high-complexity scoring document; When S < 0.6, it is classified as a low-complexity scoring document; The dynamic adjustment of sharding parameters based on document complexity, combined with the dynamically adjusted sharding parameters, splits the source document and outputs an initial text sharding sequence, specifically including: Based on the division of high-complexity or low-complexity scoring documents, dynamically adjust the initial target fragment processing parameters: target fragment length, overlap length between adjacent fragments, and implicit reference recall K value; For highly complex scoring documents, the target fragment length range is adjusted to 800~1024 character tags, the overlap length is adjusted to 128 character tags, and the implicit reference recall K value is adjusted to 5; For low-complexity scoring documents, the target fragment length range is adjusted to 300-400 character tags, the overlap length is adjusted to 32 character tags, and the implicit reference recall K value is adjusted to 2; Subsequently, a recursive character text segmentation algorithm is introduced. The target segment length and the overlap length between adjacent segments are dynamically determined based on document features. When the recursive character text segmentation algorithm detects that the segmentation position may disrupt the continuity of the context, it will trigger a recursive mechanism to backtrack and adjust the segmentation point, moving the segmentation position forward or backward to a complete semantic unit, until the recursive character text segmentation algorithm detects that the text within each segment is logically coherent. The extracted plain text information and the text associated with the virtual text fragments are segmented into initial text fragments with dynamically overlapping regions, and a set of initial text fragment sequences with unique identifiers, original document location metadata, and multimodal association tags are output. The unique identifier is the exclusive identifier of each fragment, used for subsequent fragment association retrieval. The original document location metadata records the original position of each fragment in the source document for tracing the source. The multimodal association tag is the virtual text about the fragment containing multimodal elements, marking its association relationship with the original multimodal elements. Explicit references are statements in the initial text segment that explicitly point to the target part; Implicit references are expressions that do not have an explicit reference but have a backreference relationship.

2. The financial data preprocessing method based on RAG processing encapsulation according to claim 1, characterized in that, The process of receiving the source document, performing multimodal parsing and initial processing on the source document, and outputting initial processed data specifically includes: Receive unstructured or semi-structured source documents; The source document formats include PDF, DOCX, TXT, and LaTeX; Extracting structured information from the source document and transforming it into multimodal elements: The final output contains initial processed data including plain text, virtual text fragments, and source document structure information.

3. The financial data preprocessing method based on RAG processing encapsulation according to claim 2, characterized in that, It also includes the following steps: Based on extracted document features, the corresponding multilingual combined expressions in the source document are obtained, and these expressions are further converted into a uniformly processed form. This transformation vector ensures accurate similarity comparison of semantics across different languages. Finally, cross-language citation relationships are determined, specifically including: Based on the extracted document features, the corresponding multilingual combinations in the source document are obtained, and the texts of different languages ​​in the source document are converted into vectors of a unified semantic space. Then, cross-language semantic similarity is compared based on the vectors. That is, after converting the texts of different languages ​​into vectors of a unified semantic space, the closer the meanings of the two units that are cited, the more similar the arrangement of their vector representations will be. Therefore, by comparing the similarity of the two sets of vector representations, the cross-language citation relationship can be accurately obtained.

4. The financial data preprocessing method based on RAG processing encapsulation according to claim 3, characterized in that, The step of marking explicit references, implicit references, and cross-language references according to the initial text segment sequence specifically includes: Based on the initial text segment sequence, a recognition and scanning process is performed, which more specifically includes: After identifying explicit references, the system will then use a set of predefined XML-style tag pairs, namely...<REF_EXP> and< / REF_EXP> Use these as start and end tags to precisely wrap the explicit reference statement, and add a set of target attributes as anchors in the start tag to store the target location resolved from the record; After identifying implicit references, the system uses another set of tag pairs, namely...<REF_IMP> and< / REF_IMP> The implicit reference is precisely wrapped with the start and end markers, and a globally unique temporary reference identifier is assigned to the implicit reference, which is stored in the id attribute so as to record the implicit reference using the unique identifier of the id attribute; Identifying cross-language citations, i.e., citation relationships between texts in different languages, involves additional steps after identifying cross-language citations.<REF_MLlang="en"target="Section2.1"> The reference is enclosed in a marker; After the above processing, the final output is a text segment that is precisely anchored by the reference mark.

5. The financial data preprocessing method based on RAG processing encapsulation according to claim 4, characterized in that, The text segmentation sequence based on citation tags further outputs source data with quantified confidence levels, specifically including: Receive text segments precisely anchored by reference tags, and further clarify the source content corresponding to each group of references based on the reference tags and information, and assign scores and compare the source content based on the source tracing results; When a new document shard is added, the vector index is updated in real time; When receiving with<REF_EXP> When a text segment is explicitly referenced, the structured target localization engine is activated. It uses the document structure metadata saved during the initial text segmentation to directly determine the initial segment ID corresponding to the target location pointed to by the target attribute through precise string matching and logical rules. Since the explicit reference is clearly pointed to, the tracing result has high accuracy. When receiving with<REF_IMP> When a text segment marked with `<REF_ML>` is used as an implicit reference or a text segment marked with `<REF_ML>` is used as a cross-language reference, the vectorized semantic retrieval engine is activated, and the workflow is as follows: First, all initial text segments except the current text segment are transformed into high-dimensional vectors through a set of text embedding models. The text embedding models are based on a transformer architecture and output 1024-dimensional vectors. They can be trained on large-scale general, specific domain and multilingual comparative corpora to accurately capture semantics. Secondly, the text embedding model is used to convert the sentences with citation tags in the current text segment into query vectors; Finally, based on the inverted file and product quantization-based approximate nearest neighbor search index structure, the cosine similarity between the query vector and all high-dimensional vectors is calculated to determine the top K most semantically similar candidate segments. The value of K is dynamically determined according to the document complexity. When the above is a cross-language reference, the text embedding model is replaced with the mUSE model; The confidence score calculation module is further used to assign confidence scores to the source tracing results for comparison. If an explicit reference is identified by the structured target localization engine, the score is >90. The implicit and cross-language reference candidates identified by the vectorized semantic retrieval engine are then evaluated using a nonlinear mapping function to calculate their confidence scores. This function maps cosine similarity values ​​ranging from 0 to 1 to a score range of 40 to 90. The formula only applies when the cosine similarity value is greater than 0.7; otherwise, the score is 0. The specific formula is as follows: Score=40+50*((Similarity-0.7) / (1.0-0.7)) In the formula, Similarity is the cosine similarity value. When Similarity ≤ 0.7, the score is 0. Additionally, candidate segments with cosine similarity values ​​between 0.6 and 0.7 are marked as...<FLAGtype="potential_relevant" / > , i.e., potential relevant markers; Finally, a data packet is output, which contains the current text segment ID, reference type, reference content, a list of candidate source text segment IDs, a list of corresponding confidence scores, and potential relevant tags.

6. The financial data preprocessing method based on RAG processing encapsulation according to claim 5, characterized in that, The step of performing hierarchical enhancement based on source data with quantified confidence levels and outputting enhanced text fragments specifically includes: The system receives source data with quantified confidence levels and performs automated content completion for the current text segment based on the confidence score. A more detailed hierarchical processing flow is as follows: When the highest confidence score of a certain citation is greater than or equal to the preset first threshold, the reliability of the determined candidate source text segment meets the standard. At this time, the full text of the candidate source text segment with the highest confidence is read based on the generative large language model to generate a core content summary. When synchronization is for cross-language references, a summary of the core content in the target language is generated synchronously. Insert the above core content summary into the original text segments and match it with the existing citation tags; When the highest confidence score is less than 80 points but greater than or equal to the preset second threshold, it proves that the candidate source text segment has a certain reliability but uncertainty. At this time, the three consecutive sentences containing the most relevant keywords to the quoted expression are directly extracted from the candidate source text segment with the highest confidence score to form the original text fragment. Then, the original text fragment is inserted after the existing citation mark in the format of the original text fragment, and the "uncertain="true"" mark is introduced to clarify the deviation of the "uncertain="true"" mark for subsequent modules. When the highest confidence score is less than 60 but greater than or equal to 0.6, it indicates that the candidate source text fragments are potentially relevant but have low reliability. In this case, a marker is inserted to provide corresponding prompts to subsequent modules. When the highest confidence score is less than 60, it means that no reliable relevant content can be found temporarily. Therefore, manual review tags are inserted according to the existing citation tags.

7. The financial data preprocessing method based on RAG processing encapsulation according to claim 6, characterized in that, The further verification, correction, and iterative optimization of the enhanced text segments specifically include: When it receives the enhanced text fragments, it takes the original part and the enhanced part in the text fragments as a hypothetical group, inputs it into the logical consistency evaluation model, and then compares the original content and the enhanced content and outputs a logical consistency score between 0 and 1. The closer the logical consistency score is to 1, the more logically the original content and the enhanced content match; the lower the logical consistency score, the greater the possibility of logical contradictions or irrelevance. Build a domain knowledge rule engine with a built-in scalable hard rule library defined in YAML format for specific domains. The domain knowledge rule engine supports customizing rules through a visual interface. The rules include name, trigger conditions, validation logic and error messages. When validating text segments, the domain knowledge rule engine checks against the rule base. Based on the score given by the logical consistency evaluation model and the inspection results of the domain knowledge rule engine, the iterative correction process is further activated, specifically as follows: When the logical consistency score is below 0.5, or any domain rule is violated, an iterative correction operation is initiated. That is, the confidence of the proposed group formed by the original content and the enhanced content in the text segment that caused the verification failure is reduced by 20%. Then the updated data is sent back, and the reference tracing and adaptive enhancement process is re-triggered for this proposed group. At the same time, the previous error correspondence is excluded, so as to complete the iterative correction for each group of text segments. The upper limit of the number of iterations for correction is dynamically determined based on the document complexity; Finally, when the maximum number of iterations for correction is reached and the logical consistency and rule verification are still not passed, or when the confidence score of all candidate sources is below 60 points, the text segment is marked and pushed to the manual processing platform. The step of outputting standardized encapsulation and constructing a graph based on the verified text fragments and optimized data specifically includes: First, a marking and cleaning process is carried out; Specifically, all temporary reference tags and internal tags generated during the processing will be removed, leaving only the enhanced content tags with actual functions, as well as the processing status and risk warning tags; Secondly, standardized packaging procedures are implemented; Specifically, each processed text segment is encapsulated into a structured JSON object. This JSON object serves as the information carrier and has several clearly defined fields as follows: chunk_id serves as a unique identifier for text chunks; original_text records the original text content of the text segment; augmented_content stores augmentation information, which includes the augmentation type as a summary or original text fragment, the source text segment ID, and the specific augmentation content; final_confidence is the final confidence score after multiple verifications and iterative corrections; flags record the special markers that exist; multimodal_info then associates with multimodal element information; Finally, a global fragmented association graph is constructed. During the standardized packaging process, a global fragmentation association graph in the form of a directed graph is constructed simultaneously; Each node in the global fragmented association graph corresponds to a text fragment, and each directed edge represents a verified reference relationship, that is, from the referencing text fragment to the referenced text fragment. The weight of the edge is set to the final confidence score.

8. A financial data preprocessing system for executing the financial data preprocessing method based on RAG processing encapsulation as described in claim 7, characterized in that, The financial data preprocessing system includes: The document parsing and initial analysis module is used to receive the source document, perform multimodal parsing and initial processing on the source document, and output the initial processed data. The feature extraction and adjustment module is used to further extract document features based on the initial processed data to determine the document complexity, and dynamically adjust the slicing parameters based on the document complexity. The document parsing initial segmentation module is also used to segment the source document by combining dynamically adjusted segmentation parameters and output an initial text segmentation sequence; The multilingual processing module is used to extract the document features, obtain the corresponding multilingual combination expressions in the source document, and further convert the multilingual combination expressions into a unified processing form. Then, the conversion vector ensures that the semantics of different languages ​​can be accurately compared for similarity, and finally determines the cross-language reference relationship. The semantic association identification and anchoring module is used to mark explicit references, implicit references, and cross-language references according to the initial text segment sequence. The citation tracing and confidence quantification module is used to further output tracing data with quantified confidence based on the text segment sequence with citation tags; The adaptive content enhancement module is used to perform hierarchical enhancement based on source data with quantified confidence levels and output enhanced text fragments. The verification and correction module is used to perform verification, correction, and iterative optimization on the enhanced text segments. The encapsulation and graph construction module is used to output standardized encapsulation and construct a graph based on the validated text fragments and optimized data.

Images