Data screening method based on difference comparison and related equipment thereof

Through the data screening method based on differential comparison, the target text data in financial business text data is screened out, which solves the problem of sample uncertainty and instability in the proportion of wrong examples in traditional methods, and improves the accuracy of semantic recognition methods.

CN119691144APending Publication Date: 2025-03-25PING AN TECH (SHENZHEN) CO LTD
0 Cites 0 Cited by

Patent Information

Application Number
CN202411672921.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-21
Publication Date
2025-03-25

AI Technical Summary

Technical Problem

Traditional semantic recognition methods use random sampling when building business data samples, resulting in sample uncertainty and instability in the proportion of wrong examples, which in turn affects the accuracy of the semantic analysis model.

Method used

A data filtering method based on differential comparison is proposed. By obtaining the original business text data and cleaning the data, the pre-trained semantic understanding model is input to obtain the filter probability distribution, the similarity score and difference score with the preset corpus text data are calculated, and the target text data is filtered out after sorting.

Benefits of technology

Effectively and steadily filter out target text data that are similar to the corpus text data expression in the preset corpus and have large differences in intentions, optimize the semantic understanding model, and improve the accuracy of semantic recognition methods.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention belongs to the field of natural language processing, and relates to a data screening method based on difference comparison and related equipment thereof.The method comprises the steps that to-be-screened business text data is input into a pre-trained semantic comprehension model to obtain screening probability distribution, obtaining N pieces of candidate business text data from a preset corpus text database according to the similarity scores of the business text data to be screened from large to small, obtaining candidate probability distribution of the N pieces of candidate business text data, and calculating a value score of each piece of business text data to be screened according to the screening probability distribution and the candidate probability distribution, and sorting the business text data from high to low according to the value scores, and taking a preset number of to-be-screened business text data in the top of the sorting as target text data. The invention further relates to a block chain technology, and the business text data and the related model are stored in the block chain. According to the method, the target data required by semantic comprehension model training can be stably and effectively screened out, and the efficiency of semantic comprehension model optimization is improved.
Need to check novelty before this filing date? Find Prior Art