A method and device for extracting information based on big data

By extracting text with high correlation with preset subject words from the information database, calculating emotion scores and annotating them using deep reinforcement learning models, the problems of low accuracy of emotion dictionary and high cost of model training are solved, and the accuracy of emotion analysis is improved Sex and efficiency.

CN118035444BActive Publication Date: 2024-11-29SHANGHAI BIYI NETWORK TECH CO LTD
View PDF 5 Cites 0 Cited by

Patent Information

Application Number
CN202410186792.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-20
Publication Date
2024-11-29
Estimated Expiration
2044-02-20

AI Technical Summary

Technical Problem

The existing emotion dictionary has low accuracy when the emotional polarity of the text is not obvious, and the classifier model requires a large amount of manual annotation of data, resulting in high training costs.

Method used

A method of information extraction based on big data is proposed. By extracting text with high correlation with preset subject words from the information database, calculating emotional scores and adding data annotations, using deep reinforcement learning model to train the agent and perform secondary annotation and integrate analysis to obtain emotional tendency information.

Benefits of technology

It improves the accuracy of sentiment analysis, reduces the time and cost of manual annotation, and provides high-quality annotated data for training of agile models.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118035444B_ABST
    Figure CN118035444B_ABST
Patent Text Reader

Abstract

The present invention discloses a method and device for extracting information based on big data, which relates to the field of data analysis. The method obtains a target text by calculating the correlation between text in an information database and preset keywords. The method calculates the sentiment score and annotates the data of the target text, and processes the target text separately, which helps to more accurately determine the sentiment tendency of the text, and provides high-quality annotated data for intelligent agent model training, solving the problem of long time and high cost of manual annotation. The intelligent agent performs secondary annotation on the target text with unclear sentiment polarity, which further improves the accuracy of sentiment analysis.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a method and device for extracting information based on big data. Background Art

[0002] With the rapid development of mobile Internet, information is growing exponentially. How to extract useful information from massive text data has become an important issue, and sentiment analysis technology has emerged.

[0003] Sentiment analysis is a technology that determines the emotional tendency of a text based on its content. There are two existing implementation methods: one is to implement it through artificial predefined rules and statistical analysis technology, mainly the method of sentiment dictionary; the other is to implement it through training the classifier model with labeled sentiment classification data. However, the accuracy of the sentiment dictionary will drop significantly when the sentiment polarity of the text is not particularly obvious; the classifier model requires a large amount of labeled data for model training, and the existing technology is to manually add data annotations after manually judging the sentiment of the text, which will consume a lot of time and labor costs. Summary of the invention

[0004] The purpose of the present invention is to solve the problems of low accuracy of sentiment dictionaries and high model training costs mentioned in the above background technology, and to propose a method and device for extracting information based on big data.

[0005] The first aspect of the present invention provides a method for extracting information based on big data, the method comprising:

[0006] Extracting texts with a correlation degree with a preset subject word greater than a first preset threshold from the information database and putting them into a first target data set as first target texts;

[0007] Calculate the sentiment score of each text in the first target data set and add data annotations; the data annotations include positive annotations, negative annotations and neutral annotations;

[0008] Put the first target text with data marked as neutral into the second target data set as the second target text; put the first target text with data marked as positive and negative into the third target data set as the third target text;

[0009] Using the text in the third target dataset to train a deep reinforcement learning model, an intelligent agent is obtained for judging the sentiment tendency of the text;

[0010] Perform secondary data annotation for each text in the second target data set by the agent, and put the text after the secondary data annotation into the fourth target data set;

[0011] The third target data set and the fourth target data set are integrated and analyzed to obtain the sentiment tendency information related to the preset topic in the information database.

[0012] Optionally, texts are selected from the information database in order, nouns in the texts are extracted, and the nouns are sorted according to the frequency of occurrence of each noun, and a preset number of nouns ranked at the top are used as the first type of keywords of the text;

[0013] Matching the positions of the nouns of the first type of keywords in the semantic network, taking the inverse of the path length from each noun to the preset subject word as the relevance of each noun with the preset subject word, performing weighted average on the relevance of each noun of the first type of keywords, and obtaining the relevance between the text and the preset subject word; the semantic network is a semantic network constructed based on the preset subject word;

[0014] The texts whose relevance to the preset subject words is greater than a first preset threshold value are put into the first target data set.

[0015] Optionally, selecting a first target text from the first target data set in order, and extracting adjectives and adverbs in the first target text as the second type of keywords of the first target text;

[0016] According to the preset sentiment dictionary, the second type of keywords are traversed. If the adjective is not in the sentiment dictionary, a zero score is assigned; if the adjective is in the sentiment dictionary, a corresponding sentiment score is assigned to the adjective. If there is a degree adverb before the adjective, the score is adjusted according to the degree; if there is a negative adverb before the adjective, the scoring sign is changed; the sentiment scores of all adjectives are weighted and summed to obtain the sentiment score of the first target text; the preset sentiment dictionary contains positive words and negative words, the sentiment score of the positive words is between (0, 1], and the sentiment score of the negative words is between [-1, 0);

[0017] If the sentiment score of the text is greater than the preset positive threshold, a positive sentiment annotation is added to the text; if the sentiment score of the text is less than the preset negative threshold, a negative sentiment annotation is added to the text; if the sentiment score of the text is greater than the preset negative threshold and less than the preset positive threshold, a neutral sentiment annotation is added to the text; the preset positive threshold is greater than 0, and the preset negative threshold is less than 0.

[0018] Optionally, initialize the training environment so that the agent selects actions from the action interval according to the environment state; give rewards and new environment states according to the agent's actions and environment states, so that the agent updates the strategy function according to the rewards and selects actions according to the new environment states; repeat the training steps until a preset stop condition is reached; the action interval is {-1, 1}, -1 indicates selecting negative labeling, and 1 indicates selecting positive labeling; the environment state is represented by text vectorization in a third data set; the preset stop condition includes the maximum number of training rounds and / or convergence of the convergence function.

[0019] Optionally, if the agent's action is consistent with the data annotation of the text corresponding to the environment state, a positive reward is given.

[0020] reward a =100*p

[0021] Among them, reward a is the desired positive reward, and p is the probability that the agent chooses this action;

[0022] If the agent's action selection is inconsistent with the data annotation of the text corresponding to the environment state, a negative reward will be given.

[0023] reward n =-1000*p

[0024] Among them, reward n is the desired negative reward, and p is the probability that the agent chooses this action.

[0025] Optionally, the text in the third target data set is segmented, removed, padded or truncated to obtain normalized text of equal length; the vocabulary in the normalized text is converted into a vector of fixed dimension by a word embedding method; and each vectorized text is put into a training set as an environmental state representation.

[0026] Optionally, modify the output rule of the agent to output the probability of selecting each action; determine the judgment threshold P based on the test data of the agent on the training set t ;

[0027] Vectorize each text in the second target data set and input it to the agent to obtain the probability P1 of selecting 1 and the probability P2 of selecting -1;

[0028] If P1>P2 and P1>P t , then add positive data annotation to the text; if P2>P1 and P2>P t , then add negative data annotation to the text; if P1 <P t And P2 <P t , then add neutral data annotation to the text.

[0029] A second aspect of the present invention provides a device for extracting information based on big data, the device comprising:

[0030] A text selection module, used to extract texts with a correlation degree with a preset subject word greater than a first preset threshold from the information database, and put them into a first target data set as first target texts;

[0031] A primary annotation module, used to calculate the sentiment score of each text in the first target data set and add data annotations; the data annotations include positive annotations, negative annotations and neutral annotations;

[0032] The one-time labeling module is further used to put the first target text labeled as neutral into the second target data set as the second target text; and put the first target text labeled as positive and negative into the third target data set as the third target text;

[0033] A model training module is used to train a deep reinforcement learning model using text in a third target data set to obtain an intelligent agent for judging the sentiment tendency of the text;

[0034] A secondary annotation module, used for performing secondary data annotation for each text in the second target data set through the intelligent agent, and putting the text after the secondary data annotation into the fourth target data set;

[0035] The data analysis module is used to integrate and analyze the third target data set and the fourth target data set to obtain the sentiment tendency information related to the preset topic in the information database.

[0036] Beneficial effects of the present invention:

[0037] The present invention proposes a method for extracting information based on big data, which includes: extracting texts whose correlation with preset subject words is greater than a first preset threshold from an information database, and putting them into a first target data set as first target texts; calculating the sentiment score of each text in the first target data set, and adding data annotations; the data annotations include positive annotations, negative annotations and neutral annotations; putting the first target text with data annotated as neutral into a second target data set as a second target text; putting the first target text with data annotated as positive and negative into a third target data set as a third target text; using the text in the third target data set to train a deep reinforcement learning model to obtain an intelligent agent for judging the emotional tendency of the text; performing secondary data annotation for each text in the second target data set by the intelligent agent, and putting the text after the secondary data annotation into a fourth target data set; integrating and analyzing the third target data set and the fourth target data set to obtain the emotional tendency information related to the preset subject in the information database.

[0038] The embodiment of the present invention calculates the sentiment score and annotates the data of the target text, and processes the target text separately, which helps to more accurately determine the sentiment tendency of the text, and provides high-quality annotated data for intelligent model training, solving the problems of long time and high cost of manual annotation; the target text with unclear sentiment polarity is annotated again by the intelligent agent, which further improves the accuracy of sentiment analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention will be further described below in conjunction with the accompanying drawings.

[0040] Figure 1 A flowchart of a method for extracting information based on big data is provided for an embodiment of the present invention;

[0041] Figure 2 A flowchart of another information extraction method based on big data is provided for an embodiment of the present invention;

[0042] Figure 3 A structural diagram of a news information extraction device based on big data is provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] The embodiment of the present invention provides a method for extracting news information based on big data. Figure 1 , Figure 1 A flowchart of a method for extracting information based on big data is provided in an embodiment of the present invention. The method comprises the following steps:

[0045] S101, extracting texts with a correlation degree with a preset subject word greater than a first preset threshold from an information database, and putting them into a first target data set as first target texts.

[0046] S102, calculating the sentiment score of each text in the first target data set and adding data annotations.

[0047] S103, putting the first target text with data marked as neutral into the second target data set as the second target text; putting the first target text with data marked as positive and negative into the third target data set as the third target text.

[0048] S104, using the text in the third target data set to train a deep reinforcement learning model to obtain an intelligent agent for judging the emotional tendency of the text.

[0049] S105, performing secondary data annotation for each text in the second target data set through an intelligent agent, and putting the text after the secondary data annotation into a fourth target data set.

[0050] S106, integrating and analyzing the third target data set and the fourth target data set to obtain the sentiment tendency information related to the preset topic in the information database.

[0051] Among them, data labeling includes positive labeling, negative labeling and neutral labeling.

[0052] Based on the big data-based information extraction method provided by this embodiment, the target text is processed separately by calculating the sentiment score and annotating the data of the target text, which helps to more accurately determine the sentiment tendency of the text, and provides high-quality annotated data for the training of the intelligent model, solving the problems of long time and high cost of manual annotation; the target text with unclear sentiment polarity is annotated again by the intelligent agent, which further improves the accuracy of sentiment analysis.

[0053] In one implementation, the information may be news information, which is acquired through a real-time acquisition device and stored in a database; the information may also be comment data under a related topic, which is acquired through a real-time acquisition device and stored in a database.

[0054] In one implementation, the preset subject term may be the name of a subject, such as a specific company name or a person's name; the preset subject term may also be the name of a certain field, such as computer, chemistry, etc.

[0055] In one implementation, the first preset threshold is a relevance threshold that is pre-set based on the relevance requirement and is used to filter information that is highly relevant to the preset topic, which can improve the pertinence and efficiency of the analysis and reduce the impact of irrelevant data on the analysis results.

[0056] In one implementation, the emotional tendency of the text is judged by the emotional score of each text, and then data annotation is added to the text according to the emotional tendency; the emotional tendency is divided into positive emotion, negative emotion and neutral emotion; because the emotional dictionary contains positive and negative words, the text with obvious emotional tendency can be clearly divided into positive and negative when classified, but the remaining text with unclear emotion is classified as neutral, which greatly reduces the judgment accuracy of the text with unclear emotion; therefore, the text judged as neutral by the emotional dictionary is processed separately, and a second judgment is made through the model method, thereby improving the accuracy of the emotional judgment of the text.

[0057] In one embodiment, Figure 1 Based on this, step S101 includes:

[0058] Step 1: select texts from the information database in order, extract nouns from the texts, sort them according to the frequency of occurrence of each noun, and use a preset number of nouns ranked at the top as the first type of keywords for the text.

[0059] Step 2: Match the position of each noun of the first type of keyword in the semantic network, and use the inverse of the path length from each noun to the preset subject word as the reciprocal of the path length from each noun to the preset subject word.

[0060] Step three: perform weighted averaging on the relevance of each noun of the first type of keyword to obtain the relevance between the first type of keyword and the preset subject word.

[0061] Step 4: put the texts whose relevance to the preset keywords is greater than the first preset threshold into the first target data set.

[0062] Among them, the semantic network is a semantic network constructed based on preset keywords.

[0063] In one implementation, by setting a first preset threshold of the relevance, texts that are highly relevant to the topic can be accurately screened out, which can significantly improve the pertinence and efficiency of the analysis and reduce unnecessary data processing.

[0064] In one implementation, the semantic network exists in the form of a tree, each node represents a word, and the root node is a preset keyword; the depth of each layer of nodes represents the correlation between the layer of nodes and the preset keyword, and the greater the depth, the smaller the correlation.

[0065] In one embodiment, Figure 1 Based on this, step S102 includes:

[0066] Step 1: Select a first target text from the first target data set in order, and extract adjectives and adverbs in the first target text as second-type keywords of the first target text.

[0067] Step 2: According to the preset sentiment dictionary, traverse the second type of keywords. If the adjective is not in the sentiment dictionary, assign zero score; if the adjective is in the sentiment dictionary, assign the adjective a corresponding sentiment score. If there is an adverb of degree before the adjective, adjust the score according to the degree; if there is a negative adverb before the adjective, change the scoring symbol.

[0068] Step three: weighted sum of the sentiment scores of all adjectives to obtain the sentiment score of the first target text.

[0069] Step 4: If the sentiment score of the text is greater than the preset positive threshold, a positive sentiment annotation is added to the text; if the sentiment score of the text is less than the preset negative threshold, a negative sentiment annotation is added to the text; if the sentiment score of the text is greater than the preset negative threshold and less than the preset positive threshold, a neutral sentiment annotation is added to the text.

[0070] The preset sentiment dictionary includes positive words and negative words, the sentiment scores of positive words are between (0, 1], and the sentiment scores of negative words are between [-1, 0); the preset positive threshold is greater than 0, and the preset negative threshold is less than 0.

[0071] In one implementation, by presetting positive thresholds and negative thresholds, texts with obvious emotional polarity are screened out and data annotations are added, which helps to more clearly understand the emotional tendency of the text and provides high-quality annotated data for intelligent model training, solving the problem of long time and high cost of manual annotation.

[0072] In one implementation, the preset positive threshold and the preset negative threshold are determined by the results of model training. For example, if the model finds overfitting during training and testing, the preset positive threshold is appropriately lowered and the preset negative threshold is increased.

[0073] In one embodiment, see Figure 2 ,exist Figure 1 Based on this, step S104 includes:

[0074] S1041, initializing the training environment so that the agent selects actions from the action interval according to the environment state.

[0075] S1042, giving rewards and new environmental states according to the actions of the agent and the environmental states, so that the agent updates the strategy function according to the rewards and selects actions according to the new environmental states, until a preset stop condition is reached.

[0076] Among them, the action interval is {-1, 1}, -1 indicates selecting negative labeling, and 1 indicates selecting positive labeling; the environment state is represented by the text vectorization in the third data set; the preset stop condition includes the maximum number of training rounds and / or the convergence of the convergence function.

[0077] In one implementation, the deep reinforcement learning model is trained using annotated text, and the resulting intelligent agent has the ability to judge the sentiment of the text. This model training method can fully utilize the feature extraction ability of deep learning and the decision optimization ability of reinforcement learning, thereby obtaining a more accurate and robust intelligent agent for judging the sentiment of text.

[0078] In one embodiment, Figure 2 On the basis of, step S1042 includes:

[0079] Step 1: If the action of the agent is consistent with the data annotation of the text corresponding to the environment state, a positive reward is given.

[0080] reward a =100*p

[0081] Step 2: If the action selected by the agent is inconsistent with the data annotation of the text corresponding to the environment state, a negative reward is given.

[0082] reward n =-1000*p

[0083] Among them, reward a is the desired positive reward, n is the desired negative reward, and p is the probability that the agent chooses this action.

[0084] In one implementation, the reward is determined based on the correctness and probability of the agent's choice of action, so that the agent's learning iterates in the direction of improving the certainty of its decision-making. Since the reward is related to the probability of the action, the agent may tend to explore different actions in the early stages of training, and its decisions are relatively random. As the training progresses, the agent will gradually learn to choose more accurate actions while maintaining a certain degree of exploration to avoid falling into a local optimal solution too early.

[0085] In one embodiment, Figure 2 On the basis of, before step S1041, it includes:

[0086] Step 1: segment, remove, fill or truncate the text in the third target data set to obtain normalized text of equal length.

[0087] Step 2: Convert the words in the normalized text into vectors of fixed dimension through word embedding method.

[0088] Step three: put the vectorized texts into the training set as the environment state representation.

[0089] In one implementation, by normalizing and vectorizing the text, the consistency of the input data can be ensured to facilitate model processing.

[0090] In one embodiment, Figure 1 On the basis of, step S105 comprises:

[0091] Step 1: Modify the output rule of the agent to output the probability of selecting each action.

[0092] Step 2: Determine the judgment threshold P based on the test data of the agent on the training set t .

[0093] Step three, vectorize each text in the second target data set and input it to the intelligent agent to obtain the probability P1 of selecting 1 and the probability P2 of selecting -1.

[0094] Step 4: If P1>P2 and P1>P t , then add positive data annotation to the text; if P2>P1 and P2>P t , then add negative data annotation to the text; if P1 <P t And P2 <P t , then add neutral data annotation to the text.

[0095] In one implementation, the judgment threshold P is determined based on the test data of the intelligent agent on the training set. t For example, if during the test, the minimum probability of the agent making an action selection is 0.7, then P t Set to 0.7 to improve the certainty and accuracy of sentiment judgment on text.

[0096] The embodiment of the present invention provides a news information extraction device based on big data. Figure 3 , Figure 3 A structural diagram of a news information extraction device based on big data provided by an embodiment of the present invention. The device includes:

[0097] The text selection module is used to extract texts with a correlation degree with a preset subject word greater than a first preset threshold from the information database and put them into the first target data set as the first target text.

[0098] The one-time labeling module is used to calculate the sentiment score of each text in the first target data set and add data labels. It is also used to put the first target text with data labeled as neutral into the second target data set as the second target text; and put the first target text with data labeled as positive and negative into the third target data set as the third target text.

[0099] The model training module is used to train a deep reinforcement learning model using the text in the third target data set to obtain an intelligent agent for judging the emotional tendency of the text.

[0100] The secondary annotation module is used to perform secondary data annotation for each text in the second target data set through an intelligent agent, and put the text after the secondary data annotation into the fourth target data set.

[0101] The data analysis module is used to integrate and analyze the third target data set and the fourth target data set to obtain the sentiment tendency information related to the preset topic in the information database.

[0102] Based on the big data-based information extraction device provided by this embodiment, the target text is processed separately by calculating the sentiment score and annotating the data of the target text, which helps to more accurately determine the sentiment tendency of the text, and provides high-quality annotated data for the training of the intelligent model, solving the problems of long time and high cost of manual annotation; the target text with unclear sentiment polarity is annotated again by the intelligent agent, which further improves the accuracy of sentiment analysis.

[0103] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A method for extracting information based on big data, characterized in that: The method comprises: Extracting texts with a correlation degree with a preset subject word greater than a first preset threshold from the information database and putting them into the first target data set as first target texts; the texts in the information database are news information and comment data under related topics; Calculate the sentiment score of each text in the first target data set and add data annotations; the data annotations include positive annotations, negative annotations and neutral annotations; Put the first target text with data marked as neutral into the second target data set as the second target text; put the first target text with data marked as positive and negative into the third target data set as the third target text; Using the text in the third target dataset to train a deep reinforcement learning model, an intelligent agent is obtained for judging the sentiment tendency of the text; Perform secondary data annotation for each text in the second target data set by the agent, and put the text after the secondary data annotation into the fourth target data set; Integrate and analyze the third target data set and the fourth target data set to obtain sentiment tendency information related to the preset topic in the information database; The reward calculation process of the deep reinforcement learning model includes: If the agent's action is consistent with the data annotation of the text corresponding to the environment state, a positive reward is given. Among them, reward a is the desired positive reward, and p is the probability that the agent chooses this action; If the agent's action selection is inconsistent with the data annotation of the text corresponding to the environment state, a negative reward will be given. Among them, reward n is the desired negative reward, and p is the probability that the agent chooses this action; The performing secondary data annotation for each text in the second target data set by the intelligent agent includes: Modify the output rule of the intelligent agent to output the probability of selecting each action; determine the judgment threshold P based on the test data of the intelligent agent on the training set t ; Vectorize each text in the second target data set and input it to the agent to obtain the probability P1 of selecting 1 and the probability P2 of selecting -1; If P1>P2 and P1>P t , then add positive data annotation to the text; if P2>P1 and P2>P t , then add negative data annotation to the text; if P1 <P t And P2 <P t , then add neutral data annotation to the text.

2. The information extraction method based on big data according to claim 1 is characterized in that: The extracting of texts with a correlation degree with the preset subject words greater than a first preset threshold from the information database and putting them into the first target data set as the first target texts comprises: Selecting texts from the information database in order, extracting nouns from the texts, sorting the nouns according to the frequency of occurrence of each noun, and taking a preset number of nouns ranked at the top as the first type of keywords of the text; Matching the positions of the nouns of the first type of keywords in the semantic network, taking the inverse of the path length from each noun to the preset subject word as the relevance of each noun with the preset subject word, performing weighted average on the relevance of each noun of the first type of keywords, and obtaining the relevance between the text and the preset subject word; the semantic network is a semantic network constructed based on the preset subject word; The texts whose relevance to the preset subject words is greater than a first preset threshold value are put into the first target data set.

3. The information extraction method based on big data according to claim 1 is characterized in that: The step of calculating the sentiment score of each text in the first target data set and adding data annotations includes: Selecting first target texts from the first target data set in order, extracting adjectives and adverbs in the first target texts as second-type keywords of the first target texts; According to the preset sentiment dictionary, the second type of keywords are traversed. If the adjective is not in the sentiment dictionary, a zero score is assigned; if the adjective is in the sentiment dictionary, a corresponding sentiment score is assigned to the adjective. If there is a degree adverb before the adjective, the score is adjusted according to the degree; if there is a negative adverb before the adjective, the scoring sign is changed; the sentiment scores of all adjectives are weighted and summed to obtain the sentiment score of the first target text; the preset sentiment dictionary contains positive words and negative words, the sentiment score of the positive words is between (0, 1], and the sentiment score of the negative words is between [-1, 0); If the sentiment score of the text is greater than the preset positive threshold, a positive sentiment annotation is added to the text; if the sentiment score of the text is less than the preset negative threshold, a negative sentiment annotation is added to the text; if the sentiment score of the text is greater than the preset negative threshold and less than the preset positive threshold, a neutral sentiment annotation is added to the text; the preset positive threshold is greater than 0, and the preset negative threshold is less than 0.

4. The information extraction method based on big data according to claim 1 is characterized in that: The method of using the text in the third target data set to train the deep reinforcement learning model to obtain an intelligent agent for judging the emotional tendency of the text includes: Initialize the training environment so that the agent selects an action from the action interval according to the environment state; give rewards and new environment states according to the agent's actions and environment states, so that the agent updates the strategy function according to the rewards and selects actions according to the new environment states; repeat the training steps until a preset stop condition is reached; the action interval is {-1, 1}, -1 indicates selecting a negative label, and 1 indicates selecting a positive label; the environment state is represented by a text vector in a third data set; the preset stop condition includes a maximum number of training rounds and / or convergence of a convergence function.

5. The information extraction method based on big data according to claim 4 is characterized in that: The initialization of the training environment includes: The text in the third target data set is segmented, removed, padded or truncated to obtain normalized texts of equal length; the vocabulary in the normalized texts is converted into vectors of fixed dimensions by word embedding method; and each vectorized text is put into the training set as an environmental state representation.

6. A news information extraction device based on big data, characterized in that: The device comprises: A text selection module, used to extract texts with a correlation degree with a preset subject word greater than a first preset threshold from the information database, and put them into a first target data set as first target texts; A primary annotation module, used to calculate the sentiment score of each text in the first target data set and add data annotations; the data annotations include positive annotations, negative annotations and neutral annotations; The one-time labeling module is further used to put the first target text labeled as neutral into the second target data set as the second target text; and put the first target text labeled as positive and negative into the third target data set as the third target text; A model training module is used to train a deep reinforcement learning model using text in a third target data set to obtain an intelligent agent for judging the sentiment tendency of the text; A secondary annotation module, used for performing secondary data annotation for each text in the second target data set through the intelligent agent, and putting the text after the secondary data annotation into the fourth target data set; A data analysis module, used for integrating and analyzing the third target data set and the fourth target data set to obtain sentiment tendency information related to the preset topic in the information database; The reward calculation process of the deep reinforcement learning model includes: If the agent's action is consistent with the data annotation of the text corresponding to the environment state, a positive reward is given. Among them, reward a is the desired positive reward, and p is the probability that the agent chooses this action; If the agent's action selection is inconsistent with the data annotation of the text corresponding to the environment state, a negative reward will be given. Among them, reward n is the desired negative reward, and p is the probability that the agent chooses this action; The performing secondary data annotation for each text in the second target data set by the intelligent agent includes: Modify the output rule of the intelligent agent to output the probability of selecting each action; determine the judgment threshold P based on the test data of the intelligent agent on the training set t ; Vectorize each text in the second target data set and input it to the agent to obtain the probability P1 of selecting 1 and the probability P2 of selecting -1; If P1>P2 and P1>P t , then add positive data annotation to the text; if P2>P1 and P2>P t , then add negative data annotation to the text; if P1 <P t And P2 <P t , then add neutral data annotation to the text.

Citation Information

Patent Citations

  • Method and device for obtaining response verbal skill, computer equipment and storage medium

    CN110765244A

  • A brain focus detection model establishing method and device, computer equipment and a storage medium

    CN112634255A

  • Sample data processing method and device, electronic equipment and medium

    CN114548192A

  • Stock public opinion monitoring method and system based on artificial intelligence

    CN114913012A

  • Method and apparatus for selecting annotated data, and computer device and storage medium

    WO2021139257A1