Financial text data classification method and device based on pre-training model

Abstract

The invention discloses a financial text data classification method and device based on a pre-training model, and is applied to the technical field of data classification. According to the method, firstly, financial text data are preprocessed, data containing classification identifiers are constructed, and the number of the classification identifiers is not less than one; then, a data classification model based on a pre-training model BERT is constructed, and the model is trained by utilizing the preprocessed training data through a sub-network fine tuning method; and finally, classifying the financial text data to be classified by utilizing the trained data classification model. By optimizing the fine tuning algorithm, the financial text data classification efficiency is improved, the precision of the training model is enhanced, the accuracy of the classification result is ensured, and the accuracy requirement in practical application can be met.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a method and device for classifying financial text data based on a pre-training model. Background Art

[0002] With the development of information technology, many financial basic businesses, core processes, inter-industry transactions and other affairs and activities have been running on information support carriers. The information generated in the production and operation of financial institutions has gradually been transformed into digital assets in different forms and circulated between different information networks and systems. With the in-depth application of new technologies such as big data, artificial intelligence, and cloud computing in the financial industry, data has gradually realized the transformation from information assets to production factors, and its importance has become increasingly prominent. The scope of data security threats to financial institutions has gradually expanded from within the institution to between industries, and even affected national security, social order, public interests and financial market stability.

[0003] Financial data is complex and diverse. Implementing hierarchical management of data can further clarify the objects of data protection and help financial institutions to rationally allocate data protection resources and costs. It is the basis for financial institutions to establish a sound financial data lifecycle protection framework and a prerequisite for the targeted implementation of data security management.

[0004] In this environment, financial institutions have an increasing demand for technologies that can accurately classify and process financial text data. As an advanced natural language processing (NLP) technology, the BERT (Bidirectional Encoder Representations from Transformers) model provides a new solution for the classification and processing of financial text data through its powerful language understanding and text representation capabilities. The BERT model can deeply understand the semantics and context of text data, thereby improving the accuracy and efficiency of financial text data classification. However, in the field of financial text classification, the application of the BERT model still faces some challenges and limitations. Due to the large capacity of model parameters and limited annotated data in the financial field, it may not be able to directly capture the professional terms and specific contexts in the financial field. Due to overfitting, the results on the test data are degraded, resulting in limited accuracy in financial text classification tasks. Summary of the invention

[0005] The purpose of the present invention is to provide a financial text data classification method and device based on a pre-training model in view of the above-mentioned deficiencies in the prior art.

[0006] The objective of the present invention is achieved through the following technical solution: A financial text data classification method based on a pre-training model comprises the following steps:

[0007] (1) Preprocessing the financial text data to construct data containing classification identifiers as training data, wherein the number of the classification identifiers is not less than one;

[0008] (2) Use the training data to train a data classification model based on the pre-trained model BERT. During the back propagation process, the model is trained using the sub-network fine-tuning method, including:

[0009] (2.1) Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient;

[0010] (2.2) Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters;

[0011] (2.3) Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters;

[0012] (2.4) Apply gradient mask: transform the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g ′ t =g t ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1;

[0013] (2.5) Update the first-order moment estimate: m t =β1·m t-1 +(1-β1)·g ′ t ;

[0014] (2.6) Update the second-order moment estimate: v t =β2·v t-1 +(1-β2)·g ′ t 2 ;

[0015] (2.7) Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate

[0016] (2.8) Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate

[0017] (2.9) Iterative training: Repeat steps (2.2) to (2.8) until the model converges or reaches the preset number of iterations;

[0018] (2.10) Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights;

[0019] (3) The trained data classification model is used to classify the financial text data to be classified.

[0020] Furthermore, it also includes:

[0021] The trained data classification model is deployed in the data classification system to classify the financial text data to be classified.

[0022] Furthermore, the trained data classification model is deployed in the data classification system to classify the financial text data to be classified, including:

[0023] Data upload: Through the file upload module in the user interface, the user uploads the financial text data file to be classified to the data classification system;

[0024] Data preprocessing: The system automatically preprocesses uploaded financial text data;

[0025] Model application: Use the trained data classification model to automatically classify and annotate the text data preprocessed in step (4.2);

[0026] Result storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application;

[0027] Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts based on the classification results for users to conduct in-depth analysis and decision support.

[0028] The present invention also provides a financial text data classification device based on a pre-training model, comprising:

[0029] A preprocessing module, used for constructing data containing classification identifiers as training data by preprocessing the financial text data, wherein the number of the classification identifiers is not less than one;

[0030] The model training module is used to train the data classification model based on the pre-trained model BERT using the training data. During the back propagation process, the model is trained by the sub-network fine-tuning method, including:

[0031] Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient;

[0032] Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters;

[0033] Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters;

[0034] Apply a gradient mask: Apply the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g ′ t =g t ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1;

[0035] Update first-order moment estimate: m t =β1·m t-1 +(1-β1)·g ′ t ;

[0036] Update the second moment estimate: v t =β2·v t-1 +(1-β2)·g ′ t 2 ;

[0037] Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate

[0038] Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate

[0039]

[0040] Iterative training: Repeat the above process until the model converges or reaches the preset number of iterations;

[0041] Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights;

[0042] The data classification module includes a trained data classification model, which is used to classify the financial text data to be classified.

[0043] Furthermore, it also includes:

[0044] The deployment module is used to deploy the trained data classification model to the data classification system to classify the financial text data to be classified.

[0045] Furthermore, the trained data classification model is deployed in the data classification system to classify the financial text data to be classified, including:

[0046] Data upload: Through the file upload module in the user interface, the user uploads the financial text data file to be classified to the data classification system;

[0047] Data preprocessing: The system automatically preprocesses uploaded financial text data;

[0048] Model application: Use the trained data classification model to automatically classify and annotate the text data preprocessed in step (4.2);

[0049] Result storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application;

[0050] Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts based on the classification results for users to conduct in-depth analysis and decision support.

[0051] The present invention also provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned financial text data classification method based on a pre-trained model.

[0052] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for classifying financial text data based on a pre-trained model is implemented.

[0053] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned financial text data classification method based on a pre-training model.

[0054] The beneficial effect of the present invention is that the present invention enhances the generalization ability of the model for financial field texts, enables it to better understand and process various financial text data, reduces overfitting, and its implementation method is simple and can effectively improve the accuracy of financial text data classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0056] Figure 1 This is a flow chart of a financial text data classification method based on a pre-training model provided in an embodiment of the present invention.

[0057] Figure 2 It is a flowchart of training a model through a sub-network fine-tuning method provided by an embodiment of the present invention.

[0058] Figure 3 It is a schematic diagram of a data classification system module provided by an embodiment of the present invention.

[0059] Figure 4 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0060] The present invention is described in detail below in conjunction with the accompanying drawings. In the absence of conflict, the features of the following embodiments and implementations can be combined with each other.

[0061] The present invention provides a financial text data classification method and data classification system based on a pre-training model. Figure 1 It is a flow chart of the data classification method and data classification system of the present invention, which specifically includes the following steps:

[0062] Step 1: constructing data containing classification identifiers by preprocessing the financial text data, wherein the number of the classification identifiers is not less than one;

[0063] Step 2: Use the training data to train the data classification model based on the pre-trained model BERT. During the back propagation process, the model is trained through the sub-network fine-tuning method, including:

[0064] Figure 2 This is a flowchart of training a model through a sub-network fine-tuning method provided by an embodiment of the present invention. This step is the core of the present invention and is divided into the following sub-steps:

[0065] (2.1) Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient;

[0066] (2.2) Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters;

[0067] (2.3) Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters;

[0068] (2.4) Apply gradient mask: transform the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g ′ t =g t ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1;

[0069] (2.5) Update the first-order moment estimate: m t =β1·m t-1 +(1-β1)·g ′ t ;

[0070] (2.6) Update the second-order moment estimate: v t =β2·v t-1 +(1-β2)·g ′ t 2 ;

[0071] (2.7) Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate

[0072] (2.8) Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate

[0073] (2.9) Iterative training: Repeat the above steps until the model converges or reaches the preset number of iterations;

[0074] (2.10) Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights;

[0075] Step 3: The trained data classification model is used to classify the financial text data to be classified.

[0076] In one embodiment, the method further includes: deploying the trained data classification model in a data classification system to classify the financial text data to be classified; Figure 3 As shown, the data classification system includes a file upload module, a data preprocessing module, a model classification module, a file storage module and a report generation module. Specifically, the following steps are included:

[0077] Data upload: Through the file upload module in the user interface, users can upload the financial text data files to be classified to the classification system;

[0078] Data preprocessing: The system automatically preprocesses the uploaded financial text data through the data preprocessing module;

[0079] Model application: The model classification module uses the trained financial text data classification model to automatically classify and annotate the pre-processed text data;

[0080] Result file storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application;

[0081] Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts and exportable tables based on the classification results for users to conduct in-depth analysis and decision support;

[0082] The performance comparison between the model trained using the subnetwork fine-tuning method of the present invention and the bert-base-chinese model on the financesimple dataset is shown in Table 1 below:

[0083] Table 1: Performance comparison

[0084] macro metrics precision recal‖ f1 bert-base-chinese 90.892 89.424 89.424 childNetwork-bert 92.29 91.584 91.468 Improvement 1.398 2.16 2.044

[0085] As can be seen from Table 1 above, the model method (childNetwork-BERT) trained by the subnetwork fine-tuning method of the present invention is compared with the evaluation indicators of the BERT-base-Chinese model as the baseline method. The data set contains information on listed companies of Tonghuashun from 2000 to 2022 and various research reports and company announcements collected from public channels, including 50,000 samples, each with 6 features. Compared with the baseline method, this method has improved the three evaluation indicators of macro-average precision, recall, and F1, and has practical value.

[0086] The present invention also provides a financial text data classification device based on a pre-training model, comprising:

[0087] A preprocessing module, used for constructing data containing classification identifiers as training data by preprocessing the financial text data, wherein the number of the classification identifiers is not less than one;

[0088] The model training module is used to train the data classification model based on the pre-trained model BERT using the training data. During the back propagation process, the model is trained by the sub-network fine-tuning method, including:

[0089] Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient;

[0090] Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters;

[0091] Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters;

[0092] Apply a gradient mask: Apply the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g ′ t =gt ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1;

[0093] Update first-order moment estimate: m t =β1·m t-1 +(1-β1)·g ′ t ;

[0094] Update the second moment estimate: v t =β2·v t-1 +(1-β2)·g ′ t 2 ;

[0095] Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate

[0096] Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate

[0097]

[0098] Iterative training: Repeat the above process until the model converges or reaches the preset number of iterations;

[0099] Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights;

[0100] The data classification module includes a trained data classification model, which is used to classify the financial text data to be classified.

[0101] In one embodiment, it also includes: a deployment module, which is used to deploy the trained data classification model in the data classification system to classify the financial text data to be classified.

[0102] Specifically, the process includes the following:

[0103] Data upload: Through the file upload module in the user interface, the user uploads the financial text data file to be classified to the data classification system;

[0104] Data preprocessing: The system automatically preprocesses uploaded financial text data;

[0105] Model application: Use the trained data classification model to automatically classify and annotate the text data preprocessed in step (4.2);

[0106] Result storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application;

[0107] Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts based on the classification results for users to conduct in-depth analysis and decision support.

[0108] The present invention also provides an electronic device, see Figure 4 , comprising a memory and a processor, wherein the memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above-mentioned financial text data classification method based on a pre-training model. It should be noted that, in addition to Figure 4 In addition to the memory and processor shown, the electronic device may also include other hardware according to its actual functions, which will not be described in detail.

[0109] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the program is executed by a processor, the method for classifying financial text data based on a pre-training model as described above is implemented.

[0110] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned financial text data classification method based on a pre-training model.

[0111] The computer-readable storage medium may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The computer-readable storage medium may also be any device with data processing capability, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. equipped on the device. Furthermore, the computer-readable storage medium may also include both an internal storage unit of any device with data processing capability and an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0112] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0113] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0114] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0115] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0116] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A financial text data classification method based on a pre-training model, characterized in that: The following steps are involved: (1) Preprocessing the financial text data to construct data containing classification identifiers as training data, wherein the number of the classification identifiers is not less than one; (2) Use the training data to train a data classification model based on the pre-trained model BERT. During the back propagation process, the model is trained using the sub-network fine-tuning method, including: (2.1) Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient; (2.2) Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters; (2.3) Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters; (2.4) Apply gradient mask: transform the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g′ t =g t ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1; (2.5) Update the first-order moment estimate: m t =β1·m t-1 +(1-β1)·g′ t ; (2.6) Update the second-order moment estimate: (2.7) Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate (2.8) Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate (2.9) Iterative training: Repeat steps (2.2) to (2.8) until the model converges or reaches the preset number of iterations; (2.10) Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights; (3) The trained data classification model is used to classify the financial text data to be classified.

2. The financial text data classification method according to claim 1, characterized in that: Also includes: The trained data classification model is deployed in the data classification system to classify the financial text data to be classified.

3. The financial text data classification method according to claim 2, characterized in that: Deploy the trained data classification model to the data classification system to classify the financial text data to be classified, including: Data upload: Through the file upload module in the user interface, the user uploads the financial text data file to be classified to the data classification system; Data preprocessing: The system automatically preprocesses uploaded financial text data; Model application: Use the trained data classification model to automatically classify and annotate the text data preprocessed in step (4.2); Result storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application; Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts based on the classification results for users to conduct in-depth analysis and decision support.

4. A financial text data classification device based on a pre-training model, characterized in that: include: A preprocessing module, used for constructing data containing classification identifiers as training data by preprocessing the financial text data, wherein the number of the classification identifiers is not less than one; The model training module is used to train the data classification model based on the pre-trained model BERT using the training data. During the back propagation process, the model is trained by the sub-network fine-tuning method, including: Initialization parameters: set the learning rate η, probability p, initial pre-trained model weight w0, and hyperparameters β1 and β2 of the AdamW optimizer, where β1 and β2 are exponential decay rates used to calculate the first-order and second-order moment estimates of the gradient; Calculate the gradient: In the tth iteration, calculate the gradient of the loss function corresponding to the current model parameters with respect to the parameters where w t is the model weight after the tth iteration, So w t is the loss function of the parameters; Generate sub-network gradient mask: At the tth iteration, generate a 0-1 mask matrix M t , the elements of the mask matrix are randomly drawn from a Bernoulli distribution with probability p and are used to selectively update the model parameters; Apply a gradient mask: Apply the gradient g t The mask matrix M is generated with (3.3) t Perform Hadamard product (element-by-element multiplication) to obtain the masked gradient g′ t =g t ⊙M t , so that the gradient matrix only retains the mask matrix M t The corresponding element is 1; Update first-order moment estimate: m t =β1·m t-1 +(1-p1)·g′ t ; Update the second moment estimate: Bias correction: bias correction is performed on the first-order moment estimate and the second-order moment estimate to obtain the corrected first-order moment estimate and the second moment estimate Update model weights: Update model weights based on the corrected first-order moment estimate and second-order moment estimate, as well as the learning rate Iterative training: Repeat the above process until the model converges or reaches the preset number of iterations; Fine-tuning training is completed: During the fine-tuning process, the mask matrix M t The selective update of , which realizes the update of model weights; The data classification module includes a trained data classification model, which is used to classify the financial text data to be classified.

5. The financial text data classification device according to claim 4, characterized in that: Also includes: The deployment module is used to deploy the trained data classification model to the data classification system to classify the financial text data to be classified.

6. The financial text data classification device according to claim 5, characterized in that: Deploy the trained data classification model to the data classification system to classify the financial text data to be classified, including: Data upload: Through the file upload module in the user interface, the user uploads the financial text data file to be classified to the data classification system; Data preprocessing: The system automatically preprocesses uploaded financial text data; Model application: Use the trained data classification model to automatically classify and annotate the text data preprocessed in step (4.2); Result storage: The classified and annotated data sets are stored in the system database and the designated local storage location for subsequent query, analysis and application; Report generation: Provides a user interface for users to view classification results. The system can automatically generate data analysis charts based on the classification results for users to conduct in-depth analysis and decision support.

7. An electronic device comprising a memory and a processor, characterized in that: The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement a financial text data classification method based on a pre-trained model as described in any one of claims 1-3 above.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a financial text data classification method based on a pre-training model as described in any one of claims 1 to 3 is implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, it implements a financial text data classification method based on a pre-training model as described in any one of claims 1-3.

Images