Domain data collection method and device, electronic equipment and storage medium

By performing domain relevance annotation and compliance annotation on open source corpus data, relevant and compliant data in the financial field are screened out, and corpus scarcity and data security problems in the financial field are solved, high-quality corpus data collection is achieved, and a solid foundation is provided for pre-training of large models.

CN119691157APending Publication Date: 2025-03-25CHINA MERCHANTS BANK
0 Cites 2 Cited by

Patent Information

Application Number
CN202411519964.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-29
Publication Date
2025-03-25

AI Technical Summary

Technical Problem

How to obtain a large amount of diverse, compliant and secure domain-specific professional corpus data, especially in the financial field, with fewer existing open source corpus and sensitive information.

Method used

By obtaining data from the open source corpus, performing domain relevance annotation and compliance annotation, filtering out data related to the target domain. Use the data classification model to annotate and comment data to ensure the accuracy and security of the data.

Benefits of technology

It realizes efficient extraction of professional corpus data with specific domain relevance and compliance requirements from open source corpus, ensuring the quantity and quality of data, providing a reliable data foundation for pre-training of large models in specific domains, and improving the model's understanding ability and expression accuracy in specific domains.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a field data collection method and device, electronic equipment and a storage medium, and relates to the technical field of data processing, and the method comprises the steps: obtaining an open source data set in an open source corpus, and carrying out the field correlation labeling of all data in the open source data set, and obtaining a labeled data set; screening the annotated data set according to the annotated domain correlation to obtain an annotated data subset related to the target domain; performing compliance annotation on each piece of data in the annotation data subset to obtain an annotation data subset; and screening the annotation data subset according to the compliance of the annotation to obtain target domain data. According to the method, professional corpus data which not only has financial field correlation but also meets compliance requirements is efficiently extracted from a large amount of open source corpora, the quantity and quality of data in the target field are ensured, and a reliable data basis is provided for pre-training of a large model in the financial field; therefore, the understanding ability and expression accuracy of the model in the financial field are improved.
Need to check novelty before this filing date? Find Prior Art