Data processing method and system, storage medium and electronic equipment

Abstract

The invention discloses a data processing method and system, a storage medium and electronic equipment. Comprising the steps of obtaining to-be-processed text data, and performing data type identification on the to-be-processed text data to obtain a data type result; determining a corresponding feature configuration list based on the data type result; obtaining a corresponding extraction rule according to the data type result; extracting feature data from the feature configuration list based on an extraction rule; and when the feature data meets a preset condition, generating text abstract data based on a preset abstract rule and the feature data. Through the above scheme, under a complex data structure including non-special characters and the like, feature extraction processing is performed on different data types to obtain respective corresponding feature data, so that the requirements of automatic type identification, automatic feature extraction, automatic text abstract generation and the like under the complex data structure are met, and the accuracy of obtaining the text abstract data is improved. In addition, the text abstract data is subjected to similarity algorithm recognition, so that the accuracy of a similarity calculation result is improved.

Description

technical field [0001] The present application relates to the technical field of data processing, and more specifically, to a data processing method, system, storage medium and electronic equipment. Background technique [0002] In the natural language processing task, it is judged whether two documents are similar, and the similarity degree of the two documents is calculated by the similarity algorithm. For example, when discovering microblog hot topics based on a clustering algorithm, it is necessary to measure the content similarity of each text, and then let the microblogs with sufficiently similar content be clustered into a cluster; when preprocessing the corpus, based on the text similarity, Select the duplicate text and delete it. [0003] Before the calculation of the similarity algorithm, the data is usually preprocessed. At present, the commonly used preprocessing method is to remove special characters in the data (such as punctuation, brackets, labels, etc.). Ho...

AI Technical Summary

Problems solved by technology

[0003] Before the calculation of the similarity algorithm, the data is usually preprocessed. At present, the commonly used preprocessing method is to remove special characters in the data (such as punctuation, brackets, labels, etc.). However, in the face of complex data objects (such as structure data), it will be interfered by non-special characters (letters, numbers, Chinese characters), making it impossible to select and delete repeated text in complex structured data, resulting in the generated text summary data containing repeated text, thereby reducing the Accuracy of generated text summarization data

[0004] Therefore, existing data for generating text summarization has low accuracy

Images