Generation method and generation system of public opinion data set

Abstract

The invention belongs to the technical field of public opinion data processing, and particularly relates to a public opinion data set generation method and system. The public opinion data set generation method comprises the steps that after obtained news is sent into a preprocessing module to be preprocessed, a news text is obtained and sent into an interception module; the news text is intercepted according to the enterprise entities in the interception module, and an enterprise information set of all the enterprise entities in the current news text is obtained and then sent to the feature labeling module; the feature labeling module performs feature labeling on enterprise entities based on a large language model and a text classification model, and generates public opinion data according to labeling results of the two models; the feature labeling module sends the public opinion data into the public opinion data output module; and the public opinion data output module screens corresponding public opinion data according to the training target of the downstream model to construct a public opinion data set and outputs the public opinion data set. According to the method, the high-quality public opinion data set can be efficiently generated, and the public opinion processing effect of the trained model is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of public opinion data processing, and in particular relates to a method and system for generating a public opinion dataset. Background Art

[0002] Public opinion, which spreads rapidly online, can have a significant impact on businesses and brands. Whether it's promoting positive public opinion to enhance their image and expand their visibility, or mitigating negative public opinion through timely public relations, a business or brand needs to closely monitor public opinion and take appropriate action based on the relevant information it detects. This is known as public opinion management.

[0003] Considering that arranging a large number of staff to work in shifts to identify relevant public opinions from massive amounts of news and handle public opinion is not only costly but also inefficient; therefore, more and more corporate / brand public relations departments have begun to train deep learning models with public opinion processing capabilities to handle public opinion and provide public opinion warnings in a timely manner, thereby greatly reducing labor costs.

[0004] However, in the process of training a deep learning model with the ability to handle public opinion, it is first necessary to obtain a large amount of public opinion data based on news. Technical personnel then determine the corresponding public opinion handling methods based on the public opinion data (for example, forwarding positive public opinion data, reporting negative public opinion data to the head of the public relations department, refuting rumors in negative public opinion, etc.). Finally, the public opinion data and the corresponding public opinion handling methods are combined to form a public opinion processing dataset. The public opinion processing dataset is then used to train the deep learning model to enable it to have the ability to handle public opinion. Public opinion data is basically obtained by manually screening out relevant news and extracting public opinion information from the news. In addition to the target company, public opinion information also needs to include related companies, the judgment of the company's public opinion status, the emotional color of the public opinion, etc. Therefore, manually obtaining public opinion data is not only time-consuming and labor-intensive, but also the quality of the obtained public opinion data is unstable due to different personal standards, which will directly affect the effectiveness of the trained model in handling public opinion. Summary of the Invention

[0005] The purpose of the present invention is to overcome the deficiencies of the above-mentioned prior art and provide a method for generating a public opinion dataset, which can efficiently generate a high-quality public opinion dataset and improve the public opinion processing effect of the trained model.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for generating a public opinion data set is applied to a public opinion data set generation system: the public opinion data set generation system includes a preprocessing module, a truncation module, a feature annotation module and a public opinion data output module;

[0008] The method for generating a public opinion dataset includes the following steps:

[0009] S1, sending the acquired news to the preprocessing module for preprocessing, obtaining the news text and sending it to the interception module;

[0010] S2, the interception module intercepts the news text according to the corporate entity, obtains the corporate information set of all corporate entities in the current news text and sends it to the feature annotation module;

[0011] S3, the feature annotation module performs feature annotation on enterprise entities based on the large language model and text classification model, and generates public opinion data based on the annotation results of the two models; the feature annotation module sends the public opinion data to the public opinion data output module; the feature annotation includes the annotation of sentiment polarity features, warning level features, and warning level features;

[0012] S4, the public opinion data output module filters the corresponding public opinion data according to the training objectives of the downstream model to construct a public opinion dataset and output it.

[0013] Preferably, S1 further includes the following sub-steps:

[0014] S11, the pre-processing module also includes a local database; the news is compared with the news copy stored in the local database. If the current news already exists in the local database, the current news is discarded; otherwise, the current news is sent to the local database and stored as a news copy;

[0015] S12, pre-processing the newly added news copies in the local database: unifying the language, format, and corporate entity name of the news copies to obtain news text; one news copy is converted into one news text;

[0016] S13, the pre-processing module sends the news text to the interception module.

[0017] Preferably, S12 further includes the following:

[0018] If the news copy is in a foreign language, the foreign news copy will be translated into a Chinese news copy first; after converting the Chinese news copy into HTML text, remove the tags in the HTML text, and convert all punctuation marks in the HTML text into full-width symbols, then determine whether the current HTML text contains the corporate entity name. If the current HTML text does not contain any corporate entity name, discard the current HTML text; if the current HTML text contains corporate entity names, all corporate entity names are unified into corresponding standard corporate names to obtain the corresponding news text; the local knowledge base stores standard corporate names and corporate nicknames, and one standard corporate name is bound to several corporate nicknames; if the corporate entity name in the HTML text is a standard corporate name in the local knowledge base, or the corporate entity name in the HTML text does not exist in the local knowledge base, the current corporate entity name remains unchanged; if the corporate entity name in the HTML text is a corporate nickname, the standard corporate name bound to the current corporate nickname is used as the corresponding corporate entity name.

[0019] Preferably, S2 also includes the following contents:

[0020] All sentences containing a certain enterprise entity in the current news text are extracted as the enterprise information set of the current enterprise entity in the current news text; after obtaining the enterprise information set of all enterprise entities in the current news text, it is sent to the feature annotation module.

[0021] Preferably, extracting the sentence where the business entity is located in the current news text further includes the following steps: S21, obtaining a coordinate information set of a business entity in a news text, wherein the coordinate information set includes a plurality of coordinate pairs, each coordinate pair including the starting position coordinates and the ending position coordinates of each occurrence of the business entity in the current news text;

[0022] S22, extracting the field between the sentence-ending punctuation mark to the left of the starting position coordinate in a coordinate pair and the sentence-ending punctuation mark to the right of the ending position coordinate in the current coordinate pair as a sentence where the corresponding enterprise entity is located in the current news text.

[0023] Preferably, S3 also includes the following contents:

[0024] S31, the feature annotation module includes a first annotation unit and a second annotation unit. The first annotation unit includes a pre-trained fine-tuned large language model, and the second annotation unit includes three pre-trained fine-tuned text classification models. The corporate information set of all corporate entities in the current news text is respectively fed into the four pre-trained fine-tuned models.

[0025] The large language model infers the sentiment polarity features, main and secondary organization features, and warning level features of different corporate information sets in the current news text as the labeling results of different corporate entities in the current news text;

[0026] After the three text classification models infer the sentiment polarity features, main and secondary organization features, and warning level features of the corporate information set in the current news text, they aggregate the sentiment polarity features, main and secondary organization features, and warning level features of the same corporate entity in a news text into a single annotation result;

[0027] The sentiment polarity feature includes three labels: positive, negative, and neutral; the main and secondary institution feature includes two labels: main institution and secondary institution; the warning level feature includes two labels: no warning required and warning;

[0028] S32, calculate the difference score Score of the annotation results of j news texts obtained by the first annotation unit and the second annotation unit respectively:

[0029] Where R1 is the first annotation result set, which represents all the annotation results of the first annotation unit for j news texts; R2 is the second annotation result set, which represents all the annotation results of the second annotation unit for the same j news texts; dquantity[R1, R2] represents the number of different annotation results in the first and second annotation result sets; if the feature labels of the same business entity in the same news text are not exactly the same in the first and second annotation result sets, it is recorded as a different annotation result;

[0030] quantity[R1] represents the number of annotation results in the first annotation result set; quantity[R2] represents the number of annotation results in the second annotation result set; j is a positive integer;

[0031] If the difference score 0≤Score<ε, the same annotation results in the first annotation result set and the second annotation result set are bound to the corresponding news text into a public opinion data and sent to the public opinion data output module; ε represents the first difference threshold; if the difference score Score≥ε, the current news text and the corresponding first annotation result set R1 and second annotation result set R2 are discarded.

[0032] Preferably, the public opinion dataset generation system also includes an error reporting module. If the difference score Score≥ε, the different annotation results in the first annotation result set R1 and the second annotation result set R2 are bound to the corresponding news text into error reporting data, and the error reporting data is sent to the error reporting module.

[0033] Preferably, in the feature annotation module, the training process before obtaining the four pre-trained fine-tuning models also includes the following:

[0034] Step 1: Technicians construct a first training set and a first validation set. The first training set contains n labeled data items, and the first validation set contains m labeled data items. Technicians annotate different business entities in a piece of news with sentiment polarity features, primary and secondary organization features, and warning level features, and then bind them to the corresponding news item to form a piece of labeled data. Both n and m are positive integers.

[0035] Step 2: The first training set is fed into the large language model and three text classification models for supervised training, so that the large language model learns to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity; and the three text classification models learn to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity;

[0036] Step 3: The news corresponding to each labeled data in the first validation set is fed into the three large language models and three text classification models for inference. The labeling results of different business entities in each news article obtained after inference by the large language model are recorded as the first inference result set IR1. The labeling results of different business entities in each news article obtained after inference by the three text classification models are recorded as the second inference result set IR2.

[0037] Step 4: Based on the sentiment polarity features, main and secondary organization features, and warning level features of different corporate entities in each news article in the first validation set, calculate the accuracy Accuracy1 of the first inference result set and the accuracy Accuracy2 of the second inference result set respectively. If Accuracy1 and Accuracy2 are both greater than the first accuracy threshold Accuracy0, calculate the overlap LOR of the annotation results corresponding to the same corporate entity in the same news article in the first inference result set IR1 and the second inference result set IR2:

[0038]

[0039] Where squantity[IR1,IR2] represents the number of identical labeled results in the first inference result set IR1 and the second inference result set IR2, and quantity[·] represents the number of labeled results.

[0040] When the overlap of the annotation results LOR>γ, the large language model at this time is recorded as the pre-trained fine-tuned large language model, and the three text classification models are recorded as three pre-trained fine-tuned text classification models;

[0041] If either Accuracy1 or Accuracy2 is below the first accuracy threshold Accuracy0 or the overlap of the annotation results LOR ≤ γ, the technician calculates the loss function of each model and optimizes the model in the direction of reducing the gradient of the loss function and then returns to step 1 to construct a new first training set and a first validation set to train each model in the feature annotation module.

[0042] Preferably, the large language model is the Qwen2.5 large language model, and the three text classification models are all bert-textcnn models.

[0043] The present invention also provides a public opinion dataset generation system, including: a preprocessing module, a interception module, a feature annotation module, and a public opinion data output module. Each module is programmed or configured to execute the steps of the above-mentioned method for generating a public opinion dataset.

[0044] The beneficial effects of the present invention are:

[0045] (1) The method for generating a public opinion data set of the present invention has been verified by technical personnel. The efficiency and accuracy of the obtained public opinion data are greatly improved, which reduces labor costs. Moreover, regardless of the amount of public opinion data, its overall accuracy is relatively stable.

[0046] (2) The method for generating a public opinion dataset of the present invention can not only obtain the most suitable public opinion dataset according to the training objectives of the downstream model; but also, compared with the public opinion dataset obtained manually, the public opinion dataset obtained by the present invention greatly improves the training efficiency and training effect of the downstream model.

[0047] (3) In the training process before obtaining the four pre-trained fine-tuning models, the sign of the end of training is that both the first annotation unit and the second annotation unit have a very high accuracy rate. At the same time, in the cross-validation of the first annotation unit and the second annotation unit, the annotation results obtained by the first annotation unit and the second annotation unit are highly consistent. In the use of the public opinion data set generation system, the present invention still ensures that the public opinion data obtained without human intervention is accurate by highly consistent cross-validation of the first annotation unit and the second annotation unit (the difference score between the first annotation result set R1 and the second annotation result set R2 is extremely small). BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 This is a flow chart of a method for generating a public opinion dataset of the present invention. DETAILED DESCRIPTION

[0049] In order to make the technical solution of the present invention clearer and more specific, the present invention is clearly and completely described below with reference to the accompanying drawings. Any equivalent replacement of the technical features of the technical solution of the present invention and any solution derived by conventional reasoning by ordinary technicians in this field without making any creative work shall fall within the scope of protection of the present invention.

[0050] Example 1

[0051] like Figure 1 FIG. 1 is a flow chart of a method for generating a public opinion dataset according to the present invention, which includes the following contents:

[0052] The public opinion data set generation system includes a preprocessing module, a truncation module, a feature annotation module, and a public opinion data output module.

[0053] S1, sending the acquired news to the preprocessing module for preprocessing, obtaining the news text and sending it to the interception module;

[0054] S2, the interception module intercepts the news text according to the corporate entity, obtains the corporate information set of all corporate entities in the current news text and sends it to the feature annotation module;

[0055] S3, the feature annotation module annotates the features of different corporate entities in the news text to form public opinion data, and sends the public opinion data to the public opinion data output module;

[0056] S4, the public opinion data output module filters the corresponding public opinion data according to the training objectives of the downstream model to construct a public opinion dataset and output it.

[0057] In S1, the following sub-steps are also included:

[0058] S11, the pre-processing module also includes a local database; the news is compared with the news copy stored in the local database. If the current news already exists in the local database, the current news is discarded; otherwise, the current news is sent to the local database and stored as a news copy;

[0059] S12, pre-processing the newly added news copies in the local database: unifying the language, format, and corporate entity name of the news copies to obtain news text; one news copy is converted into one news text;

[0060] S13, the pre-processing module sends the news text to the interception module.

[0061] S12 also includes the following:

[0062] If the news copy is in a foreign language, the foreign news copy will be translated into a Chinese news copy first; after converting the Chinese news copy into HTML text, the tags in the HTML text will be removed, and all punctuation marks in the HTML text will be converted into full-width symbols, and then it will be determined whether the current HTML text contains the corporate entity name. If the current HTML text does not contain any corporate entity name, the current HTML text will be discarded; if the current HTML text contains corporate entity names, all corporate entity names will be unified into corresponding standard corporate names to obtain the corresponding news text.

[0063] The local knowledge base stores standard enterprise names and enterprise nicknames, and one standard enterprise name is bound to several enterprise nicknames. If the enterprise entity name in the html text is a standard enterprise name in the local knowledge base, or the enterprise entity name in the html text does not exist in the local knowledge base, the current enterprise entity name remains unchanged. If the enterprise entity name in the html text is an enterprise nickname, the standard enterprise name bound to the current enterprise nickname is used as the corresponding enterprise entity name. When the enterprise entity name in the html text does not exist in the local knowledge base, the current enterprise entity name must also be stored in the set to be verified, and technical personnel regularly update the local knowledge base based on the set to be verified. This ensures that the enterprise entity names representing the same enterprise in subsequent public opinion data will not be varied.

[0064] S2 also includes the following:

[0065] All sentences containing a certain enterprise entity in the current news text are extracted as the enterprise information set of the current enterprise entity in the current news text; after obtaining the enterprise information set of all enterprise entities in the current news text, it is sent to the feature annotation module.

[0066] Specifically, the sentence where the enterprise entity is located in the current news text is extracted, and the following content is also included:

[0067] S21, obtaining a coordinate information set of a certain corporate entity in a news text, wherein the coordinate information set includes a plurality of coordinate pairs, each of which includes the starting position coordinates and the ending position coordinates of each occurrence of the corporate entity in the current news text.

[0068] S22, for a certain coordinate pair: extract the field between the sentence-ending punctuation mark on the left side of the starting position coordinate and the sentence-ending punctuation mark on the right side of the ending position coordinate as a sentence where the corresponding enterprise entity is located in the current news text.

[0069] Based on manual annotation experience, sentences in news articles that do not contain corporate entities contain very little public opinion information, and rarely contain any critical public opinion information; however, they consume a large amount of storage space and computing resources. S2 and its substeps can reduce redundant information in news text.

[0070] The following are also included in S3:

[0071] S31, the feature annotation module includes a first annotation unit and a second annotation unit. The first annotation unit includes a pre-trained fine-tuned large language model, and the second annotation unit includes three pre-trained fine-tuned text classification models. The corporate information set of all corporate entities in the current news text is respectively fed into the four pre-trained fine-tuned models.

[0072] The large language model is used to infer the sentiment polarity features, main and secondary organization features, and warning level features of different corporate information sets in the current news text, and obtain the labeling results of different corporate entities in the current news text; the three text classification models are used to infer the sentiment polarity features, main and secondary organization features, and warning level features of the corporate information sets in the current news text, and summarize the sentiment polarity features, main and secondary organization features, and warning level features of the same corporate entity in a news text into one labeling result.

[0073] The sentiment polarity feature includes three labels: positive, negative, and neutral; the main and secondary organization feature includes two labels: main organization and secondary organization; the warning level feature includes two labels: no warning required and warning.

[0074] Technicians can also subdivide warning level labels into: no warning required, moderate warning, high warning, etc.

[0075] Sentiment polarity features of different corporate entities in the current news text: Sentiment polarity features are labeled based on whether a corporate entity has a positive image, a negative image, or a neutral image in the current news text: Technical personnel have pre-set a set of positive and negative sentiment words. The positive sentiment word set includes positive sentiment words that are beneficial to the financial market, such as "performance growth", "stock repurchase", "profit increase", and "shareholder increase". The negative sentiment word set includes negative sentiment words that are not beneficial to the financial market, such as "stock decline", "plunge", "loss", "decline in favorability", "termination of contract", "decline in support rate", and "disgust". The pre-trained fine-tuning model infers the sentiment polarity of the corporate entity corresponding to the current corporate information set based on the frequency of occurrence of positive and negative sentiment words in the corporate information set.

[0076] The main and secondary organization characteristics of different corporate entities in the current news text: The main organization score and secondary organization score of the corresponding corporate entity in the current news text are calculated based on the number of different corporate entities in the current news text and the frequency of appearance of each corporate entity. When there is only one corporate entity in a news text, the corporate entity is the main organization; when there are only two or more corporate entities in a news text, at least one corporate entity is the main organization; a news text can contain more than one main organization and / or more than one secondary organization. The main and secondary organizations are mainly used to determine whether a certain corporate entity is the main description object in the current news text, and the secondary organization is the secondary description object in the current news text.

[0077] Warning level features for different business entities in the current news text: Based on the sentiment polarity features of different business entities in the current news text, the scores of the corresponding business entities under different warning level feature labels are calculated. For example, for a business entity, if the "positive" label in the sentiment polarity feature label scores very high, the "no warning required" label in the warning level feature label will also score very high; if the "negative" label in the sentiment polarity feature label scores very high, the "moderate warning" or "high warning" labels in the warning level feature label will also score very high.

[0078] S32, calculate the difference score Score of the annotation results of j news texts obtained by the first annotation unit and the second annotation unit respectively:

[0079]

[0080] Among them, R1 is the first annotation result set, which represents all the annotation results of the first annotation unit for j news texts; R2 is the second annotation result set, which represents all the annotation results of the second annotation unit for the same j news texts; dquantity[R1,R2] represents the number of different annotation results in the first annotation result set and the second annotation result set; if the feature labels of the same corporate entity in the same news text are not exactly the same in the first annotation result set and the second annotation result set, it is recorded as a different annotation result; quantity[R1] represents the number of annotation results in the first annotation result set; quantity[R2] represents the number of annotation results in the second annotation result set; j is a positive integer.

[0081] If the difference score 0≤Score<ε, the same annotation results in the first annotation result set and the second annotation result set are bound to the corresponding news text into a public opinion data and sent to the public opinion data output module; ε represents the first difference threshold, which is a minimum value greater than 0; if the difference score Score≥ε, the current news text and the corresponding first annotation result set R1 and second annotation result set R2 are discarded.

[0082] In this embodiment, ε=0.1.

[0083] In the present invention, when the difference score 0≤Score<ε, that is, the first annotation result set R1 and the first annotation result set R2 have only a few different annotation results, and the annotation results of the two different models in the first annotation unit and the second annotation unit are basically the same, we retain the same annotation results of the two annotation units and send them to the public opinion data output module. The difference score Score≥ε, that is, there is a large difference in the annotation results of the first annotation result set R1 and the first annotation result set R2, but without technical personnel verification, it is impossible to determine which annotation unit in the different annotation results has the correct annotation result, so this annotation result cannot be used to form public opinion data with the corresponding news text, so the current news text and the corresponding first annotation result set and second annotation result set are discarded together.

[0084] Optionally, the public opinion dataset generation system further includes an error reporting module. In S32, if the difference score Score ≥ ε, the different annotation results in the first annotation result set R1 and the second annotation result set R2 are bound to the corresponding news text to form error reporting data, and the error reporting data is sent to the error reporting module. After technical personnel regularly manually verify the newly added error reporting data in the error reporting module, they optimize the pre-trained fine-tuning model that generates truly erroneous feature labels in the error reporting data: calculate the loss function of the pre-trained fine-tuning model, and optimize the pre-trained fine-tuning model in the direction of decreasing the gradient of the loss function.

[0085] In the present invention, the error data with the difference score Score≥ε can also be collected and used to optimize the corresponding pre-trained fine-tuning model after manual verification by technical personnel. For the same news text, there are different annotation results in the first annotation result set R1 and the second annotation result set R2. It may be that there are different error labels in both annotation results, or there may be an error label in one of the annotation results. In either case, it can only be determined after manual verification by technical personnel, and after verification, the pre-trained fine-tuning model that generates the error feature label is optimized in a targeted manner. Because it is only necessary to verify the error data and optimize the corresponding model in a targeted manner based on the verified error data, this greatly reduces the workload of technical personnel in optimizing the pre-trained fine-tuning model and improves the optimization effect.

[0086] For ease of understanding, the following example illustrates the structure of a piece of public opinion data: A piece of public opinion data contains a news text T1 and a final annotation result FR, and the final annotation result FR = {R A , R B , R C}, R ARepresents the labeling result of enterprise entity A, R B Represents the labeling result of enterprise entity B, R C Represents the annotation result of the enterprise entity C, that is, the news text T1 contains three different enterprise entities, R A =(main mechanism, front, no warning required), R B =(sub-agency, neutral, no warning required), R C =(main agency, negative, high warning).

[0087] Optionally, the public opinion data in the public opinion data output module also includes the publication time of the corresponding news.

[0088] In S4, if the downstream model's training objective is to promptly detect negative public opinion about Company β and generate targeted public relations material, the public opinion data output module extracts all public opinion data related to Company β to construct a public opinion dataset. Technicians use this dataset to train the downstream model, enabling it to promptly detect negative public opinion about Company β and generate targeted public relations material.

[0089] For example, financial institutions usually set keyword alerts related to their own businesses, such as "bank run", "fraud", "loss", etc., to build a professional public opinion processing model (i.e., the downstream model mentioned in S4). These financial institutions can further improve the public opinion data set constructed by the method of the present invention based on the key alarm words to obtain a targeted training set for the public opinion processing model. After the public opinion processing model is trained in this way, the public opinion processing model can be used to conduct risk assessments on customers based on information such as customer and related company annual reports, regulatory dynamics, industry news, etc., and provide customers with professional-level public opinion warning information and investment possibility analysis reports and other functions.

[0090] In the feature annotation module, the training process before obtaining the four pre-trained fine-tuning models also includes the following steps:

[0091] In step 1, the technicians construct the first training set and the first validation set. The first training set contains n labeled data, and the first validation set contains m labeled data. The technicians label the different corporate entities in a news article with sentiment polarity features, main and secondary organization features, and warning level features, and then bind them to the corresponding news to form a labeled data. Both n and m are positive integers.

[0092] Step 2: Send the first training set to the large language model and three text classification models for supervised training, so that the large language model learns to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity; and the three text classification models learn to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity.

[0093] In this embodiment, the large language models are all Qwen2.5 large language models, and the three text classification models are bert-textcnn models.

[0094] In step 3, the news corresponding to each labeled data in the first verification set is sent to the large language model and the three text classification models for inference respectively; the labeling results of different corporate entities in each news article obtained after the large language model reasoning, including sentiment polarity features, main and sub-organization features, and warning level features, are recorded as the first inference result set IR1; the sentiment polarity features, main and sub-organization features, and warning level features of the same corporate entity in a news text obtained after reasoning by the three text classification models are summarized into a labeling result, and the labeling results of different corporate entities in each news article obtained after reasoning by the three text classification models are recorded as the second inference result set IR2.

[0095] Step 4: Based on the sentiment polarity features, main and secondary organization features, and warning level features of different corporate entities in each news article in the first validation set, calculate the accuracy Accuracy1 of the first inference result set and the accuracy Accuracy2 of the second inference result set respectively. If Accuracy1 and Accuracy2 are both greater than the first accuracy threshold Accuracy0, calculate the overlap LOR of the annotation results corresponding to the same corporate entity in the same news article in the first inference result set IR1 and the second inference result set IR2:

[0096]

[0097] Where squantity[IR1,IR2] represents the number of identical labeled results in the first inference result set IR1 and the second inference result set IR2, and quantity[·] represents the number of labeled results.

[0098] When the overlap of the annotation results LOR>γ, the large language model at this time is recorded as the pre-trained fine-tuned large language model, and the three text classification models are recorded as three pre-trained fine-tuned text classification models;

[0099] If either Accuracy1 or Accuracy2 is below the first accuracy threshold Accuracy0 or the overlap of the annotation results LOR ≤ γ, the technician calculates the loss function of each model and optimizes the model in the direction of reducing the gradient of the loss function and then returns to step 1. The technician constructs a new first training set and a first validation set to train the model in the feature annotation module.

[0100] In this embodiment, Accuracy0=99.99%, γ=0.99.

[0101] The method for generating a public opinion data set of the present invention has been verified by technical personnel. The efficiency and accuracy of the obtained public opinion data are greatly improved, and labor costs are reduced. Moreover, regardless of the amount of public opinion data, its overall accuracy is relatively stable.

[0102] The method for generating a public opinion dataset of the present invention can not only obtain the most suitable public opinion dataset according to the training objectives of the downstream model; but also, according to the long-term records of technical personnel, it is found that compared with the public opinion dataset obtained manually, the public opinion dataset obtained by using the present invention greatly improves the training efficiency and training effect of the downstream model.

[0103] In the training process before obtaining the pre-trained fine-tuning model, the sign of the end of training is that both the first labeling unit and the second labeling unit have a very high accuracy rate. At the same time, in the cross-validation of the first labeling unit and the second labeling unit, the labeling results obtained by the first labeling unit and the second labeling unit are highly overlapped. In the use of the public opinion data set generation system, the present invention still ensures that the public opinion data obtained without human intervention is accurate by highly overlapping the cross-validation of the first labeling unit and the second labeling unit (the difference score between the first labeling result set R1 and the second labeling result set R2 is extremely small). Because the pre-trained fine-tuning model types used in the first labeling unit and the second labeling unit are different, the probability of the first labeling unit and the second labeling unit having the same error in the labeling results is basically 0. Therefore, as long as there is an error in the labeling result of one of the first labeling unit and the second labeling unit, the difference score of the first labeling unit and the second labeling unit will increase.

[0104] A public opinion dataset generation system includes: a preprocessing module, a truncation module, a feature annotation module and a public opinion data output module. Each module is programmed or configured to execute the steps of the above-mentioned method for generating a public opinion dataset.

[0105] The technologies, shapes, and structures not described in detail in the present invention are all well-known technologies. It should also be pointed out that the above are only preferred embodiments of the present invention and are not intended to limit the present invention. The various components or steps in the embodiments of the present invention can be decomposed and / or recombined, and such decompositions and / or recombinations should be regarded as equivalent solutions of the present application and should fall within the scope of protection of the present invention.

Claims

1. A method for generating a public opinion dataset, applied to a public opinion dataset generation system, characterized by: The public opinion data set generation system includes a pre-processing module, an interception module, a feature annotation module and a public opinion data output module; The method for generating a public opinion dataset includes the following steps: S1, sending the acquired news to the preprocessing module for preprocessing, obtaining the news text and sending it to the interception module; S2, the interception module intercepts the news text according to the corporate entity, obtains the corporate information set of all corporate entities in the current news text and sends it to the feature annotation module; S3, the feature annotation module performs feature annotation on enterprise entities based on the large language model and text classification model, and generates public opinion data based on the annotation results of the two models; the feature annotation module sends the public opinion data to the public opinion data output module; the feature annotation includes the annotation of sentiment polarity features, warning level features, and warning level features; S4, the public opinion data output module filters the corresponding public opinion data according to the training objectives of the downstream model to construct a public opinion dataset and output it.

2. The method for generating a public opinion dataset according to claim 1, characterized in that: S1 also includes the following sub-steps: S11, the pre-processing module also includes a local database; the news is compared with the news copy stored in the local database. If the current news already exists in the local database, the current news is discarded; otherwise, the current news is sent to the local database and stored as a news copy; S12, pre-processing the newly added news copies in the local database: unifying the language, format, and corporate entity name of the news copies to obtain news text; one news copy is converted into one news text; S13, the pre-processing module sends the news text to the interception module.

3. The method for generating a public opinion dataset according to claim 2, characterized in that: S12 also includes the following: If the news copy is in a foreign language, the foreign language news copy is first translated into a Chinese news copy; after converting the Chinese news copy into HTML text, the tags in the HTML text are removed, and all punctuation marks in the HTML text are converted into full-width symbols, and then it is determined whether the current HTML text contains a corporate entity name. If the current HTML text does not contain any corporate entity name, the current HTML text is discarded; if the current HTML text contains a corporate entity name, all corporate entity names are unified into corresponding standard corporate names to obtain the corresponding news text; The local knowledge base stores standard enterprise names and enterprise nicknames. One standard enterprise name is bound to several enterprise nicknames. If the enterprise entity name in the HTML text is the standard enterprise name in the local knowledge base, or the enterprise entity name in the HTML text does not exist in the local knowledge base, the current enterprise entity name remains unchanged; If the enterprise entity name in the HTML text is an enterprise nickname, the standard enterprise name bound to the current enterprise nickname will be used as the corresponding enterprise entity name.

4. The method for generating a public opinion dataset according to claim 1, characterized in that: S2 also Includes the following: All sentences containing a certain enterprise entity in the current news text are extracted as the enterprise information set of the current enterprise entity in the current news text; after obtaining the enterprise information set of all enterprise entities in the current news text, it is sent to the feature annotation module.

5. The method for generating a public opinion dataset according to claim 4, characterized in that: Extract the sentence where the enterprise entity is located in the current news text, and also include the following content: S21, obtaining a coordinate information set of a certain corporate entity in a news text, wherein the coordinate information set includes a plurality of coordinate pairs, each of which includes the starting position coordinates and the ending position coordinates of each occurrence of the corporate entity in the current news text; S22, extracting the field between the sentence-ending punctuation mark to the left of the starting position coordinate in a coordinate pair and the sentence-ending punctuation mark to the right of the ending position coordinate in the current coordinate pair as a sentence where the corresponding enterprise entity is located in the current news text.

6. The method for generating a public opinion dataset according to claim 1, characterized in that: S3 also includes the following: S31, the feature annotation module includes a first annotation unit and a second annotation unit. The first annotation unit includes a pre-trained fine-tuned large language model, and the second annotation unit includes three pre-trained fine-tuned text classification models. The corporate information set of all corporate entities in the current news text is respectively fed into the four pre-trained fine-tuned models. The large language model infers the sentiment polarity features, main and secondary organization features, and warning level features of different corporate information sets in the current news text as the labeling results of different corporate entities in the current news text; After the three text classification models infer the sentiment polarity features, main and secondary organization features, and warning level features of the corporate information set in the current news text, they aggregate the sentiment polarity features, main and secondary organization features, and warning level features of the same corporate entity in a news text into a single annotation result; The sentiment polarity feature includes three labels: positive, negative, and neutral; the main and secondary institution feature includes two labels: main institution and secondary institution; the warning level feature includes two labels: no warning required and warning; S32, calculate the difference score Score of the annotation results of j news texts obtained by the first annotation unit and the second annotation unit respectively: Where R1 is the first annotation result set, which represents all the annotation results of the first annotation unit for j news texts; R2 is the second annotation result set, which represents all the annotation results of the second annotation unit for the same j news texts; dquantity[R1, R2] represents the number of different annotation results in the first and second annotation result sets; if the feature labels of the same business entity in the same news text are not exactly the same in the first and second annotation result sets, it is recorded as a different annotation result; quantity[R1] represents the number of annotation results in the first annotation result set; quantity[R2] represents the number of annotation results in the second annotation result set; j is a positive integer; If the difference score 0≤Score<ε, the same annotation results in the first annotation result set and the second annotation result set are bound to the corresponding news text into a public opinion data and sent to the public opinion data output module; ε represents the first difference threshold; if the difference score Score≥ε, the current news text and the corresponding first annotation result set R1 and second annotation result set R2 are discarded.

7. The method for generating a public opinion dataset according to claim 6, characterized in that: The public opinion dataset generation system also includes an error reporting module. If the difference score Score ≥ ε, the different annotation results in the first annotation result set R1 and the second annotation result set R2 are bound to the corresponding news text into error reporting data, and the error reporting data is sent to the error reporting module.

8. A method for generating a public opinion dataset according to claim 6 or 7, characterized in that: In the feature annotation module, the training process before obtaining four pre-trained fine-tuning models also includes the following: Step 1: The technician constructs a first training set and a first validation set. The first training set contains n labeled data, and the first validation set contains m labeled data. After the technicians annotate the sentiment polarity features, main and secondary organization features, and warning level features of different corporate entities in a piece of news, they bind them to the corresponding news to form a piece of annotated data, where n and m are both positive integers; Step 2: The first training set is fed into the large language model and three text classification models for supervised training, so that the large language model learns to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity; and the three text classification models learn to label the sentiment polarity features, main and secondary organization features, and warning level features of the enterprise entity; Step 3: The news corresponding to each labeled data in the first validation set is fed into the three large language models and three text classification models for inference. The labeling results of different business entities in each news article obtained after inference by the large language model are recorded as the first inference result set IR1. The labeling results of different business entities in each news article obtained after inference by the three text classification models are recorded as the second inference result set IR2. Step 4: Based on the sentiment polarity features, main and secondary organization features, and warning level features of different corporate entities in each news article in the first validation set, calculate the accuracy Accuracy1 of the first inference result set and the accuracy Accuracy2 of the second inference result set respectively. If Accuracy1 and Accuracy2 are both greater than the first accuracy threshold Accuracy0, calculate the overlap LOR of the annotation results corresponding to the same corporate entity in the same news article in the first inference result set IR1 and the second inference result set IR2: Where squantity[IR1,IR2] represents the number of identical labeled results in the first inference result set IR1 and the second inference result set IR2, and quantity[·] represents the number of labeled results. When the overlap of the annotation results LOR>γ, the large language model at this time is recorded as the pre-trained fine-tuned large language model, and the three text classification models are recorded as three pre-trained fine-tuned text classification models; If either Accuracy1 or Accuracy2 is below the first accuracy threshold Accuracy0 or the overlap of the annotation results LOR ≤ γ, the technician calculates the loss function of each model and optimizes the model in the direction of reducing the gradient of the loss function and then returns to step 1 to construct a new first training set and a first validation set to train each model in the feature annotation module.

9. The method for generating a public opinion dataset according to claim 8, characterized in that: The large language model is the Qwen2.5 large language model, and the three text classification models are all bert-textcnn models.

10. A system for generating a public opinion dataset, characterized in that: include: The preprocessing module, the interception module, the feature annotation module, and the public opinion data output module are each programmed or configured to execute the steps of a method for generating a public opinion dataset as described in any one of claims 1-9.

Images