A Chinese text sorting system based on strong encoding and Chinese word segmentation

Abstract

The present invention discloses a Chinese text sorting system based on strong encoding and Chinese word segmentation. This system implements Chinese text sorting based on a strong encoding model and Chinese word segmentation data. First, a database containing a large amount of Chinese text and corresponding labels is acquired. The labeled Chinese text data is used as input, and the Chinese text is word segmented and then encoded into a machine-readable format. The encoded sentences are then input into a Chinese text sorting model for model training. The trained model can then be used to automatically sort newly acquired Chinese text. This system achieves automated, highly accurate Chinese text sorting, taking into account the contextual relationships between Chinese words. This overcomes the low efficiency of manual text sorting and the low accuracy of traditional methods. The system is widely applicable and contributes to the intelligent development of military intelligence sorting, news topic classification, and film review classification, among other fields.

Description

Technical Field

[0001] The present invention relates to the field of text sorting, and in particular to a Chinese text sorting system based on strong coding. Background Art

[0002] Text sorting is used in many fields today, such as military intelligence sorting, news topic classification, and film review classification. With the rapid development and iteration of science and technology and society, intelligent, automated, and efficient text sorting systems are becoming a major demand. Manual tasks are not only time-consuming and inefficient, but also significantly impact the results due to human subjective emotions and work attitudes. Therefore, leveraging machine learning to implement Chinese text sorting is of great significance to the development of various sectors in Chinese society.

[0003] Although some traditional machine learning methods have achieved certain results in text sorting, with the explosive growth of data in the 21st century, the efficiency, accuracy and intelligence of traditional methods can no longer meet the needs of today's society. Therefore, it is urgent and important to invent a Chinese text sorting system with high accuracy, fast sorting speed and high intelligence. Summary of the Invention

[0004] To address the low efficiency, low accuracy, and low intelligence of current traditional methods, the present invention aims to provide a Chinese text sorting system with high accuracy, fast sorting speed, and a high level of intelligence. This system implements Chinese text sorting based on a strong encoding model and Chinese word segmentation data. First, a database containing a large amount of Chinese text and corresponding labels is acquired. The labeled Chinese text data is used as input, and the Chinese text is segmented and then encoded into a machine-readable format. The encoded sentences are then input into a Chinese text sorting model for model training. The trained model can then be used to automatically sort newly acquired Chinese text. This system achieves automated, highly accurate Chinese text sorting, overcoming the low efficiency of manual text sorting and the low accuracy of traditional methods. The system is widely applicable and can contribute to the intelligent development of military intelligence sorting, news topic classification, and film review classification, among other fields.

[0005] The objective of the present invention is achieved through the following technical solutions: an automatic target recognition system based on underwater laser images, including Chinese sorting data acquisition, a Chinese text sorting system based on strong coding and Chinese word segmentation, and a display module, wherein the Chinese text sorting system based on strong coding and Chinese word segmentation includes a Chinese text database, a preprocessing module, a Chinese text sorting modeling module, a Chinese text automatic sorting module, and a sorting result output module.

[0006] The Chinese sorting data acquisition module obtains Chinese text and labels through the Internet or actual demand tasks.

[0007] Furthermore, the Chinese text database is used to store historical Chinese text data and newly acquired Chinese text, thereby providing a data basis for the Chinese text sorting modeling module. At the same time, the module will update the newly acquired Chinese text for sorting in real time, improve the database content, and thus provide a basis for updating the model.

[0008] Furthermore, the preprocessing module is used to process Chinese text data. The only difference between the preprocessing of text in the Chinese text database and the preprocessing of newly acquired Chinese text is that the data used in the former must be divided into a training set and a validation set, so that the model effect can be verified in the modeling module and an ideal model can be obtained. In addition, the preprocessing module for Chinese text in the present invention is mainly completed by the following process:

[0009] When performing word segmentation on Chinese text, unlike English, which can directly distinguish words by spaces, if Chinese text is encoded as single characters, the relationship between the words will be ignored. In order to make full use of the prior knowledge of Chinese words, the jieba word segmenter is first used to split the Chinese text sentences into characters and words.

[0010] After dividing the sentence into characters and words, these Chinese characters and words need to be converted into digital form, that is, the encoding process, which encodes Chinese characters into a form that can be recognized by the machine. Specifically, all the words are read into a list, and the words that do not conform to the modern text structure are deleted. The frequency of each word is counted, and uncommon words with a frequency of < 2 times are deleted. Finally, in order to realize the encoding of Chinese, the remaining i-th word in the list is one-hot encoded to obtain w i =[0,0,...1,...0,0], where except for the i-th value which is 1, the rest are 0, and a 256-dimensional representation vector x corresponding to each character or word is obtained by the following formula i

[0011] x i =Ww i

[0012] Where W is the conversion matrix pre-trained with the data in the database. The position p corresponding to the i-th word i It is also a 256-dimensional vector

[0013]

[0014] The final encoded value y i =x i +p i , where the encoding dimension d model =256, c1, c2 are position control coefficients, and their values ​​are between 0 and 1.

[0015] 80% of the data is extracted from the database as the training set, and the remaining data is used as the validation set. The validation set can be used to check the recognition effect of the model.

[0016] Furthermore, the Chinese text sorting modeling module comprehensively utilizes the strong feature representation advantages of the attention model and the strong classification ability of the neural network to establish a highly accurate Chinese text sorting model. This model can automatically learn how to extract effective sentence representation features and sort based on the training set. Specifically, the encoded Chinese text and labels in the training set are input into a model composed of 6 transformer encoders to train and update the model parameters.

[0017] By observing the test results of the model in the validation set, we further modified the number of multi-heads in the self-attention to optimize the model. Finally, we obtained model C.

[0018] Furthermore, the Chinese text automatic sorting module is used to sort the Chinese text to be sorted after being processed by the preprocessing module to obtain a sorting result.

[0019] Furthermore, the sorting result output module outputs the recognition result.

[0020] The technical concept of this invention is as follows: It implements Chinese text sorting based on a strong encoding model and Chinese word segmentation data. First, a database containing a large amount of Chinese text and corresponding labels is obtained. The labeled Chinese text data is used as input, and the Chinese text is segmented and then encoded into a machine-readable format. The encoded sentences are then fed into a Chinese text sorting model for model training. The trained model can then be used to automatically sort newly acquired Chinese text. This invention achieves automated, highly accurate Chinese text sorting, overcoming the inefficiency of manual text sorting and the low accuracy of traditional methods. It is widely applicable and could contribute to the intelligentization of military intelligence sorting, news topic classification, and film review classification.

[0021] The beneficial effects of the present invention are mainly manifested in: 1. Real-time updating is achieved by updating texts through the database, thereby improving adaptability to new texts; 2. Chinese word segmentation is adopted, taking into account the meaning of Chinese words, thereby improving sorting accuracy; 3. The encoding form adopted can reduce memory usage and improve recognition efficiency; 4. The context of text is encoded through position; 5. Strong encoding modeling is adopted, which is fast and accurate. DETAILED DESCRIPTION

[0022] The present invention is further described below according to the embodiments:

[0023] The Chinese text sorting system based on strong coding and Chinese word segmentation of the present invention comprises a Chinese text database, a preprocessing module, a Chinese text sorting modeling module, a Chinese text automatic sorting module, and a sorting result output module.

[0024] The Chinese sorting data acquisition module obtains Chinese text and labels through the Internet or actual demand tasks.

[0025] The Chinese text database is used to store historical Chinese text data and newly acquired Chinese text, thereby providing a data basis for the Chinese text sorting modeling module. At the same time, the module will update the newly acquired Chinese text for sorting in real time, improve the database content, and thus provide a basis for model updates.

[0026] The preprocessing module is used to process Chinese text data. The only difference between preprocessing text in a Chinese text database and preprocessing newly acquired Chinese text is that the data used in the former must be divided into a training set and a validation set, so that the model effect can be verified in the modeling module and an ideal model can be obtained. In addition, the preprocessing module for Chinese text in the present invention is mainly completed by the following process:

[0027] When performing word segmentation on Chinese text, unlike English, which can directly distinguish words by spaces, if Chinese text is encoded as single characters, the relationship between the words will be ignored. In order to make full use of the prior knowledge of Chinese words, the jieba word segmenter is first used to split the Chinese text sentences into characters and words.

[0028] After dividing the sentence into characters and words, these Chinese characters and words need to be converted into digital form, that is, the encoding process, which encodes Chinese words into a form that can be recognized by the machine. Specifically, all words are read into a list, and words that do not conform to the modern text structure are deleted. The frequency of each word is counted, and uncommon words with a frequency of < 2 times are deleted. Finally, the remaining i-th word in the list is one-hot encoded to obtain w i =[0,0,...1,...0,0], where except for the i-th value which is 1, the rest are 0, and a 256-dimensional representation vector x corresponding to each character or word is obtained by the following formula i

[0029] x i =Ww i

[0030] Where W is the conversion matrix pre-trained with the data in the database. The position p corresponding to the i-th word i It is also a 256-dimensional vector

[0031]

[0032] The final encoded value y i =x i +p i , where the encoding dimension d model =256, c1, c2 are position control coefficients, and their values ​​are between 0 and 1.

[0033] 80% of the data is extracted from the database as the training set, and the remaining data is used as the validation set. The validation set can be used to check the recognition effect of the model.

[0034] The Chinese text sorting modeling module comprehensively utilizes the strong feature representation advantages of the attention model and the strong classification capabilities of the neural network to establish a highly accurate Chinese text sorting model. This model can automatically learn how to extract effective sentence representation features and sort based on the training set. Specifically, the encoded Chinese text and labels in the training set are input into a model consisting of 6 transformer encoders to train and update the model parameters.

[0035] By observing the test results of the model in the validation set, we further modified the number of multi-heads in the self-attention to optimize the model. Finally, we obtained model C.

[0036] The Chinese text automatic sorting module is used to sort the Chinese text to be sorted after being processed by the preprocessing module to obtain a sorting result.

[0037] The sorting result output module outputs the recognition results.

[0038] The above embodiments are used to illustrate the present invention rather than to limit the present invention. Any modifications and changes made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A Chinese text sorting system based on strong encoding and Chinese word segmentation, characterized by: It includes Chinese text database, preprocessing module, Chinese text sorting modeling module, Chinese text automatic sorting module, and sorting result output module; The Chinese text database is used to store historical Chinese text data and newly acquired Chinese texts. At the same time, the module updates the newly acquired Chinese texts for sorting in real time to improve the database content. The preprocessing module is used to process Chinese text data, including the following steps: (A) First, the jieba word segmenter is used to split the Chinese text sentences into characters and words; (B) Convert these Chinese characters and words into digital form, specifically: read all the words into a list, delete the words that do not conform to the modern character structure, count the frequency of each word, delete the uncommon words with a frequency of less than 2 times, and finally, use one-hot encoding to get w for the remaining i-th word in the list. i =[0,0,...1,...0,0], where except for the i-th value which is 1, the rest are 0, and a 256-dimensional representation vector x corresponding to each character or word is obtained by the following formula i x i =Ww i Among them, W is the conversion matrix pre-trained with the data in the database; the position p corresponding to the i-th word i It is also a 256-dimensional vector The final encoded value y i =x i +p i , where the encoding dimension d model =256, c1, c2 are position control coefficients, and their values ​​are between 0 and 1; Extract 80% of the data from the database as the training set, and the remaining data as the validation set to check the recognition effect of the model; The Chinese text sorting modeling module automatically learns how to extract effective sentence representation features and sort them based on the training set. Specifically, the Chinese text and labels in the encoded training set are input into a model composed of 6 transformer encoders to train and update the model parameters. By observing the test results of the model in the validation set, we further modify the number of multi-heads in the self-attention to optimize the model; finally, we obtain model C. The Chinese text automatic sorting module is used to sort the Chinese text to be sorted after being processed by the pre-processing module to obtain a sorting result; The sorting result output module outputs the recognition result.

Images