News text automatic classification method based on multi-segmentation feature fusion and CNN (Convolutional Neural Network)

Abstract

The invention discloses a news text automatic classification method based on multi-segmentation feature fusion and CNN, and relates to the technical field of data processing. The method comprises the steps of obtaining a news text and performing parallel word segmentation through multiple word segmentation tools to generate multiple word segmentation sets and unregistered word sets; counting the number of unlisted words and calculating word segmentation complexity, and obtaining word segmentation precision of each word segmentation mode under different news categories in combination with historical data; according to word segmentation complexity and word segmentation precision, a classification computing power coefficient is dynamically configured, and the number of CNNs allocated to all word segmentation features is determined; in a pre-trained text classification network group, corresponding networks are called according to the distribution number to classify respective word segmentation sets, and the category with the highest occurrence frequency is selected as a final classification result through an integration decision. According to the method, the problems of difficulty in unlisted word recognition and word segmentation ambiguity caused by a single word segmentation mode are effectively solved, and the accuracy and reliability of news text classification are improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an automatic news text classification method based on multi-word feature fusion and CNN. Background Technology

[0002] News text segmentation and classification is a key technology in the fields of natural language processing and information retrieval. It aims to automatically analyze and categorize massive amounts of news data using computer algorithms, assigning it to predefined topic categories. Efficient and accurate segmentation and classification technology can improve the efficiency of information organization and retrieval, meeting users' needs for quickly obtaining target information.

[0003] Most existing technologies employ a single word segmentation tool or fixed strategy to process input text. However, due to the dynamic evolution of news language, new vocabulary, professional terms, and internet slang are constantly emerging. The inherent dictionaries and rules of a single word segmenter are insufficient to fully adapt to the complex and ever-changing linguistic phenomena in news texts. This leads to failure in identifying out-of-vocabulary words and word segmentation ambiguity, resulting in errors in the segmentation results. For example, it may fail to correctly segment newly emerging proper nouns or domain terms, or produce ambiguous segmentations in different contexts, affecting classification accuracy. Summary of the Invention

[0004] This invention addresses the technical problems in existing news classification technologies, such as difficulty in identifying out-of-vocabulary words, word segmentation ambiguity, and low classification accuracy due to reliance on a single word segmentation strategy. It provides an automatic news text classification method based on multi-word segmentation feature fusion and CNN.

[0005] The technical solution of the present invention to solve the above-mentioned technical problems is as follows: This invention provides an automatic news text classification method based on multi-word segmentation feature fusion and CNN, including: The news text to be classified is obtained, and multiple word segmentation methods are used to process the words, resulting in multiple word sets. Then, multiple sets of out-of-vocabulary words are obtained. Based on multiple sets of out-of-vocabulary words, multiple segmentation complexities are obtained through analysis, and the segmentation accuracy of the various segmentation methods for multiple news categories is obtained. Configure multiple classification computing power coefficients based on multiple word segmentation complexities and multiple word segmentation accuracies; Based on multiple classification computing power coefficients and multiple word segmentation sets, the news text is classified and identified to obtain classification results.

[0006] The beneficial effects of this invention are: Compared to existing technologies, this invention firstly overcomes the inherent limitations of single word segmentation tools by employing multiple word segmentation methods in parallel processing of text. This allows for the capture of text features from different dimensions, reducing errors caused by out-of-vocabulary words and segmentation ambiguity. Secondly, by quantitatively analyzing the segmentation complexity and historical segmentation accuracy of each segmentation result, differentiated reliability assessments are assigned to different segmentation sources. Thirdly, based on this assessment, classification computing power coefficients are dynamically configured, achieving adaptive allocation of computing resources and enabling more comprehensive model discrimination of highly reliable segmentation features. Finally, by fusing classification results generated by multiple convolutional neural networks based on different word segmentation sets, an ensemble strategy is used to derive the final classification, thereby improving the overall accuracy and robustness of news text classification. Attached Figure Description

[0007] Figure 1 This is a flowchart illustrating the automatic news text classification method based on multi-word feature fusion and CNN provided by the present invention. Figure 2 This is a logical diagram illustrating the automatic news text classification method based on multi-word feature fusion and CNN provided by the present invention. Detailed Implementation

[0008] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0009] In the description of this invention, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the stated features. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0010] In the description of this invention, the term "for example" is used to mean "used as an example, illustration, or description." Any embodiment described as "for example" in this invention is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use the invention. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that the invention can be made without using these specific details. In other instances, well-known structures and processes will not be described in detail to avoid obscuring the description of the invention with unnecessary detail. Therefore, the invention is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed herein.

[0011] Example 1, as Figure 1 , Figure 2 As shown, this embodiment of the invention provides an automatic news text classification method based on multi-word segmentation feature fusion and CNN, including: S10: Obtain the news text to be classified, perform word segmentation using multiple word segmentation methods, obtain multiple word sets, and process to obtain multiple sets of out-of-vocabulary words; Specifically, the news text to be categorized is obtained, and multiple word segmentation methods are used to process the words, resulting in multiple word sets. These sets are then processed to obtain multiple sets of out-of-vocabulary words, including: Obtain the news text to be categorized; Multiple word segmentation methods are used for word segmentation to obtain multiple word segmentation sets, where each word segmentation set includes multiple words; Each word segmentation set is sieved to obtain multiple sets of out-of-vocabulary words.

[0012] First, the news text to be categorized is obtained. This news text is the raw news text data received through a specified data source or input interface, including text information such as news title, body content, publication time, and source. For example, it can be web page text crawled from a news website, news information obtained through an application programming interface, or text records read from a local database, which serves as the starting input for subsequent word segmentation and automatic categorization processes.

[0013] Secondly, multiple word segmentation methods are employed to process the acquired news text, resulting in multiple word segmentation sets. Specifically, different word segmentation tools or algorithms are simultaneously invoked to independently analyze the news text. Each segmentation method outputs a set of all words segmented by that method, called a word segmentation set. Multiple segmentation methods are processed in parallel, ultimately generating multiple word segmentation sets corresponding to the number of segmentation methods. Each word segmentation set contains multiple words produced by that particular method. Word segmentation methods refer to text processing engines based on different dictionaries and segmentation rules. Through different segmentation principles and strategies, they can produce differentiated segmentation results for the same news text, thereby obtaining multiple complementary word segmentation views to more comprehensively cover various vocabulary and linguistic phenomena that may exist in the news text.

[0014] Furthermore, each obtained word segmentation set is cross-referenced and filtered to obtain multiple sets of out-of-vocabulary (OV) words. An OV set refers to a subset of words that exist in a word segmentation set generated by a specific segmentation method but do not appear in word segmentation sets generated by any other segmentation method. The words in this OV set represent segmentation results unique to a specific segmentation tool or algorithm, and may include emerging words not widely included, technical terms, domain-specific terms, or ambiguous segmentation fragments, reflecting the differences between this segmentation method and other methods when processing the current text.

[0015] Specifically, this screening process compares each word in each segmentation set with the words in all other segmentation sets. Words that appear in one segmentation set but not in any of the others are identified as out-of-vocabulary (OV) words for that segmentation method. The sum of all identified OV words constitutes the OV word set corresponding to that segmentation method. By traversing all segmentation sets, multiple OV word sets are ultimately obtained, equal to the number of segmentation sets.

[0016] Specifically, each word segmentation set is filtered to obtain multiple sets of out-of-vocabulary words, including: Randomly select the first word segmentation set, and search for multiple first words in the first word segmentation set in multiple other word segmentation sets. The first words that are not retrieved are taken as the first out-of-vocabulary words, and the first out-of-vocabulary word set is obtained. Continue to filter other word sets to obtain multiple sets of out-of-vocabulary words.

[0017] First, one of the generated word segmentation sets is randomly selected as the first word segmentation set. Each word segmentation unit in the first word segmentation set is used as a search term, and a comprehensive search and comparison is performed on the set of all other word segmentation sets. When a word segmentation unit exists in the first word segmentation set but is not found in any of the other word segmentation sets, that word segmentation unit is determined to be a first out-of-vocabulary (OV) word. All first OV words that meet this condition together form the first OV word set.

[0018] Secondly, after constructing the first set of out-of-vocabulary (OV) words, the same screening process is applied to the remaining word segments sequentially. For each word segmentation set to be processed, its internal words are cross-validated with all other word segments to filter out the OV words unique to that word segmentation set. By traversing all word segments, multiple OV word sets are finally established, with the same number of OV words as the original word segments and corresponding one-to-one.

[0019] S20: Based on multiple sets of out-of-vocabulary words, analyze and obtain multiple word segmentation complexities, and obtain multiple word segmentation accuracies for multiple news categories using the various word segmentation methods; Specifically, based on multiple sets of out-of-vocabulary words, multiple segmentation complexities are analyzed and obtained, and the segmentation accuracy of the various segmentation methods for multiple news categories is acquired, including: Count the number of out-of-vocabulary words in multiple out-of-vocabulary word sets; Calculate the ratio of the number of multiple out-of-vocabulary words to the maximum number of out-of-vocabulary words, and use this as the multiple word segmentation complexities; The segmentation accuracy of the various segmentation methods for multiple news categories is obtained.

[0020] First, the number of out-of-vocabulary (OV) words in each obtained OV set is statistically analyzed to calculate the number of OV words in multiple OV sets. Second, the segmentation complexity is quantified based on these OV word counts. Specifically, the maximum value among all OV word counts is determined, and the ratio of each OV word count to this maximum value is calculated. This ratio reflects the proportion of a particular segmentation method's segmentation result relative to the most complex case when processing the current text. The calculated result is defined as the segmentation complexity corresponding to that segmentation method. In essence, a higher segmentation complexity value indicates more specific segmentations produced by the segmentation method, lower consensus between its segmentation results and other methods, and a correspondingly higher potential segmentation uncertainty or error.

[0021] Furthermore, the segmentation accuracy of various word segmentation methods under different news categories is obtained. This process is achieved by analyzing historical word segmentation records. For each word segmentation method, the proportion of segmentation results that have been verified as correct by manual or authoritative standards when segmenting different categories of news text in the past time period is statistically analyzed. This proportion represents the word segmentation accuracy of that word segmentation method under a specific news category, reflecting the accuracy and reliability of the word segmentation tool in different semantic domains.

[0022] Specifically, the segmentation accuracy of the various segmentation methods for multiple news categories is obtained, including: Obtain a set of multiple historical segmentation records for news texts using various segmentation methods over a historical period; Obtain the percentage of historical segmentation records that have been verified to be accurate from multiple historical segmentation record sets, and use this percentage as multiple segmentation accuracy values.

[0023] First, all word segmentation records generated from processing various categories of news text using different word segmentation methods throughout history are obtained, forming multiple historical word segmentation record sets. Each historical word segmentation record set specifically corresponds to all word segmentation outputs of a particular word segmentation method within a specific news category.

[0024] Secondly, the accuracy of each historical word segmentation record set is evaluated. By comparing each word segmentation record in the historical word segmentation record set with the correct word segmentation results verified by manual annotation or authoritative standards, the number of word segmentation records judged to be accurate is counted, and the proportion of the number of accurate records in the total number of records in the corresponding historical word segmentation record set is calculated. This proportion is defined as the word segmentation accuracy of this word segmentation method in a specific news category.

[0025] For example, suppose we need to evaluate the historical segmentation accuracy of a dictionary-matching-based segmentation method within the specific category of technology news. First, we extract the segmentation records of 1000 technology news texts processed by this method, forming a historical segmentation record set. Second, we compare each of these 1000 segmentation records with the correct segmentation results verified by manual annotation or authoritative standards. For instance, the original sentence is "Artificial intelligence is reshaping business models," and the standard segmentation should be "AI / positive / reshaping / business models," while the actual output of this segmenter is "artificial / intelligent / positive / reshaping / business / model."

[0026] If, out of 1000 records, 850 records have segmentation results that completely match the standard results or are judged to be correctly segmented as keywords, then the number of accurate records is 850. The segmentation accuracy of this method in technology news is 850 / 1000 = 85%.

[0027] This word segmentation accuracy quantifies the historical performance of different word segmentation tools in different subject areas, and can provide a basis for judging the reliability of each word segmentation result in the current classification task.

[0028] S30: Configure multiple classification computing power coefficients based on multiple word segmentation complexities and multiple word segmentation precisions; Specifically, based on multiple segmentation complexities and multiple segmentation accuracies, multiple classification computing power coefficients are configured, including: Based on multiple word segmentation complexities, multiple first-class computing power coefficients are calculated and determined; Multiple word segmentation accuracies are used as multiple secondary classification computing power coefficients; Calculate and configure multiple basic category computing power coefficients based on multiple first category computing power coefficients and multiple second category computing power coefficients; Calculate the ratio of multiple segmentation precisions to multiple segmentation complexities, and use it as a multiple segmentation reliability coefficient; Multiple discrete coefficients of multiple word segmentation reliability coefficients are calculated separately, and multiple basic classification computing power coefficients are compensated to obtain multiple classification computing power coefficients.

[0029] First, the computational power coefficient for the first classification is calculated based on the word segmentation complexity. Word segmentation complexity reflects the uncertainty of the segmentation result. A higher complexity indicates that the segmented word set produced by the current segmentation method contains more out-of-vocabulary words or ambiguous segments, resulting in relatively low reliability. The computational power coefficient for the first classification is a fundamental parameter used to regulate the allocation of computational resources in subsequent classification stages, enabling differentiated processing of segmentation results with varying reliability. Specifically, the computational power coefficient for the first classification = 1 - word segmentation complexity. A higher word segmentation complexity value indicates more specific segments and greater uncertainty in the results produced by the segmentation method, leading to lower reliability. By subtracting this word segmentation complexity value, the computational power coefficient for the first classification is reduced accordingly as the word segmentation complexity increases. This proactively limits the scale of computational resources invested in high-complexity, low-reliability segmentation results, effectively suppressing the potential negative impact of low-quality segmentation features on overall classification accuracy during the integrated classification process and reducing the interference of unreliable segmentation results on the overall classification decision.

[0030] Simultaneously, the segmentation accuracy of each word segmentation method under different news categories is directly used as the second classification computational power coefficient. Segmentation accuracy reflects the historical reliability of the word segmentation tool in a specific domain; higher accuracy requires more computational resources for the corresponding classification process. The second classification computational power coefficient is an important indicator for measuring the historical performance stability of the word segmentation tool, used to prioritize the in-depth mining and full verification of high-precision word segmentation results when allocating computational resources. This second classification computational power coefficient directly inherits the reliability characteristics represented by segmentation accuracy, ensuring that in subsequent classification processes, word segmentation methods with high accuracy verified by historical data can obtain more sufficient model computational resources, thereby enhancing the contribution and influence of high-quality word segmentation features in the final classification decision.

[0031] Furthermore, the computational power coefficients of the first and second categories are fused to calculate the basic category computational power coefficient. Specifically, the basic category computational power coefficient = first category computational power coefficient × second category computational power coefficient. The first category computational power coefficient reflects the immediate reliability of the current text's specific word segmentation result, and its value is negatively correlated with the word segmentation complexity; the second category computational power coefficient reflects the historical accuracy of the word segmentation tool under the corresponding news category. The multiplicative fusion method ensures that the basic category computational power coefficient can only reach the ideal level when both coefficients maintain high values. This calculation mechanism avoids the bias that may arise from relying on only a single dimension and effectively suppresses the resource allocation obtained by word segmentation results that perform poorly in any dimension, thereby achieving reasonable pre-allocation of computing resources.

[0032] For example, when the computational power coefficient of the first category of a word segmentation method is 0.8 and the computational power coefficient of the second category is 0.9, its basic category computational power coefficient is 0.72; while when the computational power coefficient of the first category of another word segmentation method is 0.4 and the computational power coefficient of the second category is 0.9, its basic category computational power coefficient is only 0.36. This result is in line with the expected resource allocation, that is, when historical performance is similar, less computing resources are allocated to the results with lower reliability in the current word segmentation.

[0033] Furthermore, the ratio of segmentation accuracy to segmentation complexity is calculated as the segmentation reliability coefficient. This segmentation reliability coefficient characterizes the relative stability of each segmentation method under the premise of considering the current text complexity. The higher the segmentation reliability coefficient, the better the segmentation quality can be maintained in the current complex text environment.

[0034] Finally, the basic classification computing power coefficient is optimized and compensated by analyzing the distribution characteristics of each word segmentation reliability coefficient. Specifically, the relative deviation of each word segmentation reliability coefficient from the average of all word segmentation reliability coefficients is calculated. Relative deviation = (word segmentation reliability coefficient - average of all word segmentation reliability coefficients) / average of all word segmentation reliability coefficients. This relative deviation is used as a compensation factor, and the final optimized classification computing power coefficient is obtained by calculating (1 + relative deviation) × basic classification computing power coefficient.

[0035] For example, suppose there are three word segmentation methods, and their calculated basic classification computing power coefficients are: Method A 0.6, Method B 0.8, and Method C 0.4. The corresponding word segmentation reliability coefficients are: Method A 1.2, Method B 1.5, and Method C 0.9.

[0036] The mean reliability coefficient is (1.2 + 1.5 + 0.9) / 3 = 1.2. The relative deviation of method A is (1.2 - 1.2) / 1.2 = 0; the relative deviation of method B is (1.5 - 1.2) / 1.2 = 0.25; and the relative deviation of method C is (0.9 - 1.2) / 1.2 = -0.25. The classification computing power coefficients are: method A = (1 + 0) × 0.6 = 0.6; method B = (1 + 0.25) × 0.8 = 1.0; and method C = (1 - 0.25) × 0.4 = 0.3.

[0037] Through this compensation mechanism, method B, with its significantly higher word segmentation reliability coefficient than the average level, receives a higher final computing power allocation; method C, with its lower word segmentation reliability coefficient than the average level, receives a correspondingly lower computing power allocation; and method A, with its word segmentation reliability coefficient at the average level, receives an unchanged computing power allocation, thus reflecting the principle of dynamic resource adjustment based on relative performance.

[0038] Specifically, when all word segmentation reliability coefficients are similar, the deviation approaches 0, indicating high consistency in the evaluation results across all dimensions of the current text, and the basic classification computing power coefficient remains stable. When a word segmentation reliability coefficient is significantly higher than the mean, its positive deviation will generate a compensation coefficient greater than 1, thereby improving the final computing power allocation corresponding to that word segmentation method. This compensation mechanism can keenly identify word segmentation methods that perform exceptionally well in specific text environments and strengthen their advantages through dynamic weighting.

[0039] Conversely, if the reliability coefficient of a certain word segmentation is significantly lower than the mean, its negative deviation will generate a compensation coefficient of less than 1, thereby reducing its computing power allocation. By adopting an adaptive compensation strategy based on the dispersion of the group, the adaptability of the computing power allocation scheme to special text scenarios is effectively improved, ensuring that when facing complex language phenomena, classification resources can be tilted towards the most reliable word segmentation features, thereby enhancing the robustness and accuracy of the overall classification system.

[0040] S40: Based on the multiple word segmentation sets, classify and identify the news text according to multiple classification computing power coefficients to obtain classification results.

[0041] Specifically, based on the multiple word segmentation sets and according to multiple classification computing power coefficients, the news text is classified and identified to obtain classification results, including: Based on multiple classification computing power coefficients and the preset number of text classification networks, the number of multiple text classification networks is calculated respectively. Within a pre-configured text classification network cluster based on CNN, multiple text classification networks are invoked, and multiple word segmentation sets are input separately to obtain multiple network classification result sets. The network classification result that appears most frequently in multiple network classification result sets is selected as the classification result.

[0042] First, based on the finalized classification computing power coefficients, a proportional allocation calculation is performed using the preset total number of text classification networks. Each classification computing power coefficient is multiplied by the preset total number of text classification networks to obtain the specific number of text classification networks that should be allocated to each word segmentation set, ensuring that highly reliable word segmentation features receive more network resources for in-depth analysis. Specifically, since the sum of the quantities after direct multiplication usually exceeds the upper limit of the total number, normalization processing is required: each classification computing power coefficient is divided by the sum of all classification computing power coefficients to obtain the weight ratio of each word segmentation set, and then this weight ratio is multiplied by the preset total number of text classification networks. The calculation result is rounded to the nearest integer to obtain the number of text classification networks.

[0043] The preset total number of text classification networks is a pre-defined total number of convolutional neural network instances available for parallel processing. This number is determined based on a combination of available computing resources and real-time classification requirements, such as a fixed value of 20 or 50 depending on the server's GPU memory capacity or expected response time. This preset total number of text classification networks serves as an upper limit for computing resource allocation, ensuring that the sum of all allocated networks does not exceed the system's maximum capacity.

[0044] For example, assuming the preset text classification network has a total of 20 elements, the final classification computing power coefficients for the three segmentation methods are 0.6, 1.0, and 0.3, respectively. First, the sum of the classification computing power coefficients is calculated as 0.6 + 1.0 + 0.3 = 1.9. Next, the weight ratio of each segmentation method is calculated: the first segmentation method is 0.6 / 1.9 ≈ 0.316, the second is 1.0 / 1.9 ​​≈ 0.526, and the third is 0.3 / 1.9 ≈ 0.158. Then, the theoretical allocation number is calculated: the first segmentation method is 0.316 × 20 = 6.32, the second is 0.526 × 20 = 10.52, and the third is 0.158 × 20 = 3.16. The results are rounded to the nearest integer: the first segmentation method allocates 6, the second allocates 11, and the third allocates 3. At this point, the total number is 6 + 11 + 3 = 20, which exactly meets the preset requirement for the total number of text classification networks. This allocation scheme can accurately match the system's carrying capacity and reflects the principle of prioritizing high computing power coefficients.

[0045] Secondly, within the pre-constructed convolutional neural network-based text classification network cluster, a corresponding number of independent text classification networks are invoked for each word segmentation set according to the calculated network allocation scheme. These text classification networks are fully trained deep learning models used to map the input word segmentation sequence features to news category probability distributions, establishing a non-linear mapping from the text feature space to the category label space. They can automatically learn and identify deep semantic patterns of different news categories. Each invoked text classification network receives its corresponding word segmentation set as input features, calculates and outputs a classification result through forward propagation, and the classification results of all invoked text classification networks collectively constitute the network classification result set.

[0046] Specifically, the configuration steps for the text classification network group include: Based on historical news text classification data, multiple sample word sets and a set of sample news categories with classification labels were collected. Based on CNN, construct multiple text classification networks; The training and testing data are randomly divided into multiple sample word sets and sample news category sets. Multiple text classification networks are trained and tested separately until convergence is achieved, thus obtaining a text classification network group.

[0047] First, prepare the basic data required for model training. Based on historical news text classification data, collect multiple sample word sets and a set of sample news categories that have been accurately classified and labeled. Historical news text classification data refers to original news articles from a specific past time period and their corresponding manually verified category labels. This historical time range is determined comprehensively based on the news cycle and data stability requirements that the model needs to cover; for example, news data from the most recent three full years or the most recent 50,000 consecutively added news records. This news text classification data should cover all target news categories and reflect the text distribution characteristics in real-world scenarios.

[0048] Secondly, multiple initial text classification networks are constructed based on a convolutional neural network (CNN) architecture. A CNN is a deep feedforward neural network that includes convolutional computation layers. Building multiple text classification networks based on this architecture can effectively capture local semantic features and key patterns in text. Each text classification network includes word embedding layers, convolutional layers, pooling layers, and fully connected classification layers, possessing a complete mapping capability from the original word segmentation sequence to the final category determination.

[0049] Furthermore, a strategy of repeated sampling and cross-validation is employed for model training and evaluation. Specifically, random partitioning is repeatedly performed on multiple sample word sets and corresponding sample news category sets to randomly divide the data into training and testing data. Multiple text classification networks are trained and tested separately until all text classification networks reach convergence criteria. This results in a text classification network cluster composed of multiple fully trained and differentiated text classification networks, providing a reliable model foundation for subsequent ensemble classification. The convergence criteria are set based on the model's performance stability on the validation set; for example, if the improvement in classification accuracy is less than 0.5% over 10 consecutive training epochs, the text classification network is considered to have reached convergence.

[0050] For example, since there is a complex nonlinear semantic relationship between news text features and category labels, and convolutional neural networks have outstanding advantages in local feature extraction and hierarchical pattern learning, a convolutional neural network architecture is chosen to construct a text classification network.

[0051] Specifically, this text classification network mainly consists of an input layer, a feature extraction layer, and a classification output layer. The input layer receives a vectorized sequence of word segments, which is mapped into a dense vector representation by a word embedding layer. The feature extraction layer employs a multi-scale one-dimensional convolutional kernel structure, with kernel widths set to 3, 4, and 5, each width configured with 128 filters to capture local semantic features of different ranges. Each convolutional layer is followed by a ReLU activation function and a max-pooling layer, with the pooling window covering the entire sequence length. The classification output layer concatenates the multi-channel pooled features and maps them through a fully connected layer, finally outputting the probability distribution of each category through a Softmax function.

[0052] During training, key hyperparameters included a learning rate of 0.001, a maximum of 100 training epochs, and a batch size of 64. The learning rate was set to balance model convergence stability and training efficiency; the number of training epochs ensured the model fully learned text semantic patterns; and the batch size balanced memory consumption and gradient update stability. Specifically, a supervised learning approach was adopted. Random partitioning was repeatedly performed from multiple collected sample word segments and sample news category sets, with each partition generating training and test subsets in an 8:2 ratio. The word segments from the training subset were used as input, and the corresponding news categories were used as supervision signals. The network parameters were iteratively updated using backpropagation and the Adam optimizer. The cross-entropy loss function was used to measure the difference between the predicted category distribution and the true labels, and model performance was continuously monitored using the test subset.

[0053] When the improvement in classification accuracy is less than 0.5% over 10 consecutive training epochs, the text classification network is considered to have reached convergence. By independently executing the above training process on multiple initialized text classification networks, a text classification network cluster consisting of multiple fully trained and differentiated models is finally obtained, providing a multi-perspective discriminative basis for subsequent ensemble classification.

[0054] In the pre-trained text classification network group, the number of text classification networks assigned to each word segmentation set is calculated according to the aforementioned steps, and the corresponding number of text classification networks are invoked. Each independent word segmentation set is input into its assigned text classification network group, and each text classification network performs forward computation on that word segmentation set and outputs a category determination result. Finally, the category results generated by all invoked text classification networks are aggregated to form the network classification result set.

[0055] Next, statistical aggregation is performed on all output category labels in the network classification result set. By precisely counting the frequency of each category label in the entire result set, its proportion of the total number of results is calculated. Finally, the category label with the highest frequency and the largest proportion is selected as the final classification result for the news text.

[0056] In summary, this integrated decision-making mechanism can effectively combine the discriminative information learned by multiple text classification networks from different word segmentation features, thereby improving the accuracy and reliability of classification conclusions.

[0057] In summary, the embodiments of this application have at least the following technical effects: Compared to existing technologies, this application firstly constructs a parallel processing framework by integrating multiple word segmentation tools, effectively overcoming the limitations of a single word segmentation method and improving the ability to identify out-of-vocabulary words and resolve ambiguities. Secondly, it innovatively establishes a dual evaluation mechanism for word segmentation complexity and historical accuracy, providing a reliable basis for subsequent processing. Thirdly, based on a resource allocation strategy using dynamic computing power coefficients, it achieves intelligent scheduling of computing resources, ensuring that high-reliability features are fully mined. Finally, through the ensemble decision-making of convolutional neural network groups, it integrates the discriminative information of multi-dimensional word segmentation features, significantly improving the accuracy and robustness of news text classification while ensuring processing efficiency.

[0058] In summary, through multi-level technological innovation, a complete technology chain has been formed, from word segmentation optimization and resource allocation to integrated classification, providing a practical and effective solution for the automatic classification of large-scale news texts.

[0059] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this specification. Additionally, the processes depicted in the accompanying drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are possible or may be advantageous.

[0060] The above description is only a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0061] This specification and accompanying drawings are merely illustrative examples of this application and are intended to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from its scope. Therefore, if such modifications and modifications fall within the scope of this application and its equivalents, this application intends to include such modifications and modifications.

Claims

1. An automatic news text classification method based on multi-word segmentation feature fusion and CNN, characterized in that, The method includes: The news text to be classified is obtained, and multiple word segmentation methods are used to process the words, resulting in multiple word sets. Then, multiple sets of out-of-vocabulary words are obtained. Based on multiple sets of out-of-vocabulary words, multiple segmentation complexities are obtained through analysis, and the segmentation accuracy of the various segmentation methods for multiple news categories is obtained. Configure multiple classification computing power coefficients based on multiple word segmentation complexities and multiple word segmentation accuracies; Based on multiple classification computing power coefficients and multiple word segmentation sets, the news text is classified and identified to obtain classification results.

2. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 1, characterized in that, The news text to be categorized is obtained, and multiple word segmentation methods are used to process the words, resulting in multiple word sets. Further processing yields multiple sets of out-of-vocabulary words, including: Obtain the news text to be categorized; Multiple word segmentation methods are used for word segmentation to obtain multiple word segmentation sets, where each word segmentation set includes multiple words; Each word segmentation set is sieved to obtain multiple sets of out-of-vocabulary words.

3. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 2, characterized in that, Each word segmentation set is filtered to obtain multiple sets of out-of-vocabulary words, including: Randomly select the first word segmentation set, and search for multiple first words in the first word segmentation set in multiple other word segmentation sets. The first words that are not retrieved are taken as the first out-of-vocabulary words, and the first out-of-vocabulary word set is obtained. Continue to filter other word sets to obtain multiple sets of out-of-vocabulary words.

4. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 1, characterized in that, Based on multiple sets of out-of-vocabulary words, multiple segmentation complexities are obtained through analysis, and the segmentation accuracy of the various segmentation methods for multiple news categories is obtained, including: Count the number of out-of-vocabulary words in multiple out-of-vocabulary word sets; Calculate the ratio of the number of multiple out-of-vocabulary words to the maximum number of out-of-vocabulary words, and use this as the multiple word segmentation complexities; The segmentation accuracy of the various segmentation methods for multiple news categories is obtained.

5. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 4, characterized in that, Obtain the segmentation accuracy of the various segmentation methods for multiple news categories, including: Obtain a set of multiple historical segmentation records for news texts using various segmentation methods over a historical period; Obtain the percentage of historical segmentation records that have been verified to be accurate from multiple historical segmentation record sets, and use this percentage as multiple segmentation accuracy values.

6. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 1, characterized in that, Based on multiple segmentation complexities and multiple segmentation accuracies, multiple classification computing power coefficients are configured, including: Based on multiple word segmentation complexities, multiple first-class computing power coefficients are calculated and determined; Multiple word segmentation accuracies are used as multiple secondary classification computing power coefficients; Calculate and configure multiple basic category computing power coefficients based on multiple first category computing power coefficients and multiple second category computing power coefficients; Calculate the ratio of multiple segmentation precisions to multiple segmentation complexities, and use it as a multiple segmentation reliability coefficient; Multiple discrete coefficients of multiple word segmentation reliability coefficients are calculated separately, and multiple basic classification computing power coefficients are compensated to obtain multiple classification computing power coefficients.

7. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 1, characterized in that, Based on multiple classification computing power coefficients and the multiple word segmentation sets, the news text is classified and identified separately to obtain classification results, including: Based on multiple classification computing power coefficients and the preset number of text classification networks, the number of multiple text classification networks is calculated respectively. Within a pre-configured text classification network cluster based on CNN, multiple text classification networks are invoked, and multiple word segmentation sets are input separately to obtain multiple network classification result sets. The network classification result that appears most frequently in multiple network classification result sets is selected as the classification result.

8. The automatic news text classification method based on multi-word feature fusion and CNN according to claim 7, characterized in that, The configuration steps for the text classification network group include: Based on historical news text classification data, multiple sample word sets and a set of sample news categories with classification labels were collected. Based on CNN, construct multiple text classification networks; The training and testing data are randomly divided into multiple sample word sets and sample news category sets. Multiple text classification networks are trained and tested separately until convergence is achieved, thus obtaining a text classification network group.

Images