NLP-based research data standardization method and system

Abstract

The invention belongs to the field of data processing, and particularly relates to an NLP-based research data standardization method and system. The method comprises the following steps: a standard library manufacturing step: creating a standard library, formulating standard descriptions of various research fields and storing the standard descriptions in the standard library, the standard descriptions comprising research field standard descriptions, index standard descriptions and index option standard descriptions; a data input step: inputting survey data and description information of the project, the description information including field description, index description and index option description; an information identification step: carrying out word segmentation on the input description information, then carrying out vectorization to obtain word vectors, carrying out identification on the word vectors and a standard library by utilizing LSTM, and selecting a preset number of standard descriptions with the highest similarity from the calculated similarities; and an association confirmation step: selecting the most accurate one from the identified standard descriptions, and associating the most accurate one with the corresponding description information. According to the method, the processing accuracy can be ensured while the processing efficiency is ensured.

Description

technical field [0001] The invention belongs to the field of data processing, and in particular relates to an NLP-based survey data standardization method and system. Background technique [0002] Market research is the use of scientific methods to purposefully and systematically collect, record, and organize information and materials related to marketing, analyze market conditions, understand the current situation and development trends of the market, and provide objective and correct information for market forecasts and marketing decisions. data process. With the continuous development and changes of the times, the scope of market research is no longer limited to marketing companies, but has expanded to government departments, enterprises and institutions, etc. [0003] However, in order for market research to play a role, the volume of research data must pass the test. If the volume of research data is too small, there will be a problem that the research is not comprehen...

AI Technical Summary

Problems solved by technology

The manual processing is identified and judged by the staff. The staff need to remember all the standardized research fields, indicators, and standard descriptions of the index options, and then analyze the research data and index descriptions and standard descriptions of a single project to select the most suitable one. Associating with the standard description not only has very high requirements on the staff but also has a huge workload, and it is easy to make mistakes

The machine processing method is automatically associated by the system through semantic recognition. Although the efficiency is guaranteed, errors are prone to occur when a certain item or indicator has multiple standard descriptions with similar semantics

Images