Source collection strategy optimization method and device for public opinion monitoring source library

Abstract

The invention provides an information source acquisition strategy optimization method and device for a public opinion monitoring information source library. The method comprises the following steps: acquiring information source feature information in the public opinion monitoring information source library; determining a state vector of a public opinion monitoring information source library according to the information source feature information; inputting the state vector of the public opinion monitoring information source library into a strategy network to obtain an information source acquisition decision, and executing the information source acquisition decision to obtain original data of a target acquisition information source; calculating a reward value of the target acquisition information source according to the original data of the target acquisition information source; constructing an information source experience group; calculating a training convergence index of the strategy network according to the information source experience group; according to the training convergence index of the policy network, updating training parameters of the policy network; according to the method, the model is optimized by combining deep learning and public opinion acquisition, an intelligent acquisition scheme which adapts to the dynamic nature of the information source, balances the acquisition efficiency and value and improves the decision accuracy is determined, and technical support is provided for improvement of public opinion monitoring information source management and acquisition efficiency collaboratively.

Description

Technical Field

[0001] This invention belongs to the fields of information detection and artificial intelligence technology, and in particular relates to a method and apparatus for optimizing the information source collection strategy of a public opinion monitoring information source database. Background Technology

[0002] In the field of public opinion monitoring, the collection and management of information source data is the core link in achieving effective public opinion analysis. With the explosive growth of internet information volume, the types of information sources are becoming increasingly diversified, covering various channels such as news media, social platforms, industry forums, and self-media accounts. There are significant differences in the information release frequency, content authority, and dissemination influence of different information sources.

[0003] Traditional methods for collecting public opinion monitoring information sources often employ fixed rule-driven models or rely on manually configured keywords for targeted crawling. These methods exhibit several limitations: they cannot adapt to the dynamic changes in the value of information sources, struggle to capture sudden high-value sources in a timely manner, and waste computing and storage resources on low-value sources that have not provided effective information for a long period. Furthermore, traditional methods lack a quantitative feedback mechanism for collection effectiveness, and the optimization of collection decisions depends on manual experience, resulting in insufficient timeliness and accuracy, making it difficult to meet the needs of large-scale, highly dynamic public opinion monitoring.

[0004] In summary, there is an urgent need for a source management method that can achieve adaptive optimization of data acquisition decisions in order to solve the problems existing in traditional methods. Summary of the Invention

[0005] This invention provides a method for optimizing the source collection strategy of a public opinion monitoring source database. By combining deep learning with public opinion collection to optimize the strategy network, an intelligent collection scheme is determined that adapts to the dynamic nature of information sources, balances collection efficiency and value, and improves decision-making accuracy. This provides technical support for the coordinated improvement of public opinion monitoring source management and collection efficiency. The method for optimizing the source collection strategy of this public opinion monitoring source database includes:

[0006] Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold:

[0007] Obtain source characteristic information from the public opinion monitoring source database, including the historical average value of the source, the time of source collection, the source category, the success rate of source collection, and the content of the source;

[0008] Based on the source characteristic information, determine the state vector of the public opinion monitoring source database;

[0009] The state vector of the public opinion monitoring source database is input into the policy network to obtain the source collection decision. The source collection decision is executed to obtain the raw data of the target source. The source collection decision includes the target source and the collection method. The policy network is trained on a deep learning network based on the historical state vector of the public opinion monitoring source database.

[0010] Based on the raw data collected from the target information source, a reward value is calculated for the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision.

[0011] Construct a source experience group; the source experience group includes the state vector of the public opinion monitoring source database, source collection decisions, source raw data, and source reward value;

[0012] Based on the information source experience set, the training convergence index of the policy network is calculated; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network.

[0013] Update the training parameters of the policy network based on the training convergence metric of the policy network.

[0014] This invention provides a source acquisition strategy optimization device for a public opinion monitoring source database. By combining deep learning with public opinion acquisition to optimize the strategy network, an intelligent acquisition scheme is determined that adapts to the dynamic nature of information sources, balances acquisition efficiency and value, and improves decision-making accuracy. This provides technical support for the coordinated improvement of public opinion monitoring source management and acquisition efficiency. The source acquisition strategy optimization device for the public opinion monitoring source database includes:

[0015] Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold:

[0016] The information acquisition module is used to acquire source characteristic information from the public opinion monitoring source database. The source characteristic information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content.

[0017] The state vector determination module is used to determine the state vector of the public opinion monitoring source database based on the source feature information.

[0018] The data acquisition module is used to input the state vectors of the public opinion monitoring source database into the strategy network to obtain source acquisition decisions, and execute the source acquisition decisions to obtain the raw data of the target source; the source acquisition decisions include the target source and the acquisition method; the strategy network is trained on a deep learning network based on the historical state vectors of the public opinion monitoring source database.

[0019] The reward value calculation module is used to calculate the reward value of the target information source based on the raw data collected from the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision.

[0020] The information source experience group construction module is used to construct information source experience groups; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, information source raw data, and information source reward value;

[0021] The training convergence index calculation module is used to calculate the training convergence index of the policy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network.

[0022] The training parameter update module is used to update the training parameters of the policy network based on the training convergence metric of the policy network.

[0023] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-mentioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0024] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0025] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0026] In this embodiment of the invention, source feature information from the public opinion monitoring source database is obtained. This source feature information includes the historical average value of the source, the time of source collection, the source category, the success rate of source collection, and the source content. Based on the source feature information, a state vector of the public opinion monitoring source database is determined. The state vector of the public opinion monitoring source database is input into a policy network to obtain a source collection decision. Executing the source collection decision yields the raw data of the target source. The source collection decision includes the target source and the collection method. The policy network is trained on a deep learning network based on the historical state vector of the public opinion monitoring source database. Based on the raw data of the target source, a reward value for the target source is calculated. This reward value is used for... The invention evaluates the value of public opinion information corresponding to information source collection decisions; constructs an information source experience group; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, original information source data, and information source reward values; calculates the training convergence index of the strategy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the strategy network; and updates the training parameters of the strategy network based on the training convergence index of the strategy network. This embodiment of the invention optimizes the strategy network by combining deep learning with public opinion collection, determining an intelligent collection scheme that adapts to the dynamics of information sources, balances collection efficiency and value, and improves decision-making accuracy, providing technical support for the coordinated improvement of public opinion monitoring information source management and collection efficiency. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0028] Figure 1 This is a flowchart of the method for optimizing the information source collection strategy of the public opinion monitoring information source database in this embodiment of the invention;

[0029] Figure 2 This is a specific example diagram illustrating the acquisition of raw data from the target information source in an embodiment of the present invention;

[0030] Figure 3 This is a specific example diagram illustrating the calculation of the reward value of the target information source in an embodiment of the present invention;

[0031] Figure 4 This is a specific example diagram illustrating the training convergence metric of the computational policy network in an embodiment of the present invention;

[0032] Figure 5 This is a structural example diagram of the information source acquisition strategy optimization device for the public opinion monitoring information source database in an embodiment of the present invention;

[0033] Figure 6 This is a structural diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0034] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0035] The acquisition, transmission, storage, use, and processing of data in the technical solution of this invention all comply with relevant regulations.

[0036] It should be noted that in the embodiments of the present invention, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of the present invention. However, they do not mean that the applicant has used or necessarily used the solution.

[0037] As mentioned earlier, in the existing technology, the management of public opinion monitoring source databases mainly adopts a source evaluation and recommendation scheme based on multi-dimensional feature fusion. By extracting features such as content, site, and link of the source, a pre-trained machine learning model is used to score them, and then the source database is added, deleted, and new sources are recommended based on the scores. However, this scheme has problems such as static lag of the evaluation model, inability to adapt to environmental changes, lack of closed-loop feedback mechanism, disconnect between evaluation results and actual monitoring effects, suboptimal resource allocation, and difficulty in balancing source exploration and utilization.

[0038] To address this problem, the inventors discovered that deep learning theory can be applied to information source database management. By constructing an interaction system between a decision-making agent and the public opinion environment, information source database management is modeled as a deep learning decision-making process. An instant reward value based on intelligence value is designed, and a deep neural network strategy network and experience playback training loop are adopted. This allows the agent to learn the optimal collection strategy through continuous interaction, thereby achieving adaptive management of the information source database and improving the capture rate of high-value intelligence and resource utilization efficiency.

[0039] Figure 1 This is a flowchart of the method for optimizing the information source collection strategy of the public opinion monitoring information source database in an embodiment of the present invention, such as... Figure 1 As shown, the optimization method for the information source collection strategy of this public opinion monitoring information source database includes:

[0040] Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold:

[0041] Step 101: Obtain source characteristic information from the public opinion monitoring source database. The source characteristic information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content.

[0042] Step 102: Determine the state vector of the public opinion monitoring source database based on the source characteristic information;

[0043] Step 103: Input the state vector of the public opinion monitoring source database into the strategy network to obtain the source collection decision, and execute the source collection decision to obtain the original data of the target source; the source collection decision includes the target source and the collection method; the strategy network is obtained by training a deep learning network based on the historical state vector of the public opinion monitoring source database;

[0044] Step 104: Calculate the reward value of the target information source based on the raw data collected from the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision.

[0045] Step 105: Construct a source experience group; the source experience group includes the state vector of the public opinion monitoring source database, source collection decisions, source raw data, and source reward value;

[0046] Step 106: Calculate the training convergence index of the policy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network.

[0047] Step 107: Update the training parameters of the policy network according to the training convergence index of the policy network.

[0048] As shown in Figure 1, in this embodiment of the invention, the following steps are repeated until the training convergence index of the policy network reaches a preset threshold to optimize the source acquisition strategy of the public opinion monitoring source database: Acquire source feature information, including the historical average value of the source, the source acquisition time, the source category, the source acquisition success rate, and the source content; determine the state vector of the public opinion monitoring source database based on the source feature information; input the state vector into the policy network trained with historical state vectors to obtain a source acquisition decision containing the target source and acquisition method, and execute this decision to obtain the original data of the target source; calculate the reward value for evaluating the value of public opinion information based on the original data; construct a source experience group containing the state vector, source acquisition decision, original data, and reward value; calculate the training convergence index based on the source experience group to quantify the deviation between the predicted output and the target output of the policy network; update the training parameters of the policy network based on the training convergence index.

[0049] Compared with existing information source evaluation and recommendation schemes based on multi-dimensional feature fusion, which extract information source features, train offline models for scoring, and manage information sources according to scores, this approach models information source database management as a deep learning decision-making process, constructs reward values ​​based on intelligence value, and adopts a policy network trained by deep learning networks and a training and update mechanism driven by information source experience groups. This enables adaptive optimization of information source collection strategies, thereby solving the problems of static lag, lack of closed-loop feedback, and suboptimal resource allocation in existing evaluation models, and improving the capture rate of high-value public opinion intelligence and the efficiency of collection resource utilization.

[0050] In step 101, source feature information is obtained from the public opinion monitoring source database. The source feature information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content.

[0051] In a specific embodiment, the information source characteristic information obtained from the public opinion monitoring information source database includes the historical average value of information sources, the time of information source collection, the type of information source, the success rate of information source collection, and the content of information sources, including:

[0052] Source historical average value acquisition: Obtain the source value corresponding to the past N times the source was collected, calculate the arithmetic mean of these source values, and use it as the historical average value of the source, reflecting the stability of the source's past output of high-value intelligence.

[0053] Source acquisition time: retrieve the system timestamp of the last time the source was acquired and record it as the source acquisition time. Calculate the time interval between the current time and the timestamp to help measure the freshness of the source.

[0054] Source category acquisition: Based on the inherent attributes of the source, the source is classified and determined, including types such as technical forums, news media, developer communities, code hosting platforms, and social media accounts. One-hot encoding is used to represent these types and form source category features.

[0055] Source acquisition success rate: Statistically analyze the execution results of the most recent N acquisition tasks for this source, calculate the ratio of successful acquisitions to the total number of acquisitions, and obtain the source acquisition success rate. Successful acquisition is defined as successfully acquiring the source content and completing the parsing. Failure includes situations such as server unresponsiveness and broken links.

[0056] Source content acquisition: Targeted crawling of information sources to obtain the original content such as recently published articles and posts. The original content is then processed using an NLP model to extract content topic vectors, which serve as information source content features, reflecting the recent content orientation and relevance of the information source.

[0057] In this embodiment, determining the state vector of the public opinion monitoring source database based on source characteristic information may include:

[0058] Based on the source category, the source category state vector is obtained through one-hot coding;

[0059] Based on the source content, the source content state vector is obtained through a natural language processing model;

[0060] Based on the historical average value of the information source, the state vector of the historical average value of the information source is obtained through normalization.

[0061] Based on the source acquisition success rate, the source acquisition success rate state vector is obtained through normalization processing;

[0062] Based on the source acquisition time, the source acquisition time state vector is obtained through normalization processing.

[0063] In a specific embodiment, the source category state vector is generated as follows: the source category can be divided into 5 categories, namely technical forums, news media, developer communities, code hosting platforms, and social media accounts, and is processed using one-hot encoding. For example, the technical forum corresponds to the vector [1, 0, 0, 0, 0], the news media corresponds to [0, 1, 0, 0, 0], and so on. The encoding result is directly output as the source category state vector.

[0064] Source content state vector generation: The recent original content of the source is input into the natural language processing model. The model performs semantic encoding on the text, extracts high-dimensional semantic features, and outputs a dense vector with uniform dimension. This vector is the source content state vector, which is used to represent the orientation and relevance of the source content.

[0065] Source historical average value state vector generation: The range of the source historical average value is [-0.1, 20], which corresponds to the reward of -0.1 for no value collection to 20 points for high-risk events. The Min-Max normalization method is used to map this value to the interval [0, 1] to obtain the source historical average value state vector, which has a 1-dimensional dimension.

[0066] Source acquisition success rate state vector generation: The range of the source acquisition success rate is [0, 1], that is, 0 represents complete failure and 1 represents complete success. This value is directly used as the normalization result to form the source acquisition success rate state vector, which has a 1-dimensional dimension.

[0067] Source acquisition time state vector generation: The source acquisition time is represented by the time interval between the current time and the last acquisition time. The maximum effective time interval is set to 72 hours. The Min-Max normalization method is used to map the time interval value to the interval [0, 1]. The longer the interval, the closer the value is to 1. When the interval is 0, the value is 0. The source acquisition time state vector is obtained, and the vector dimension is 1.

[0068] State vector concatenation and integration: The state vectors corresponding to the above-mentioned source categories, source content, historical average value of the source, source acquisition success rate, and source acquisition time are concatenated in sequence to form a complete state vector for a single source. The state vector of the public opinion monitoring source database is a matrix composed of the complete state vectors of all sources arranged in rows, with each row corresponding to the comprehensive state characteristics of a source.

[0069] Figure 2 This is a specific example diagram illustrating the acquisition of raw data from the target information source in an embodiment of the present invention, as shown below. Figure 2 As shown, the state vector of the public opinion monitoring source database is input into the policy network to obtain the source collection decision. Executing the source collection decision yields the raw data of the target source, which may include:

[0070] Step 201: Input the state vector of the public opinion monitoring source database into the policy network, and determine the source collection decision through a greedy algorithm:

[0071] Step 202: Execute the information source acquisition decision, obtain the text, image or video information of the target information source, and obtain the raw data of the target information source after format parsing.

[0072] In a specific embodiment, the state vector is input into the policy network to obtain the source acquisition decision and is executed to obtain the raw data of the target acquisition source:

[0073] Determining information source acquisition decisions based on policy networks and greedy algorithms:

[0074] In this embodiment, the strategy network adopts a deep Q network (DQN). The state matrix of the public opinion monitoring source database, that is, the comprehensive state vector of each source corresponding to each row, is input into the strategy network. The network outputs the estimated long-term cumulative reward (Q value) for the collection action corresponding to each source.

[0075] The ε-greedy algorithm is used for decision-making: the initial value of ε is set to 0.3, and during implementation, it gradually decreases to 0.05 with training iterations. The source with the largest Q value is selected as the target source with a probability of 1-ε. The collection method is set to targeted crawling, that is, priority is given to collecting the recently updated content of the source. The source is randomly selected with a probability of ε and a full scan collection method is used to realize the exploration of unknown high-value sources.

[0076] The final output of the information source collection decision is clear: for example, the target information source is Security Technology Forum B, and the collection method is to specifically crawl the content containing the keywords of the Aegis system published in the past 72 hours.

[0077] Execute the decision and analyze the obtained raw data:

[0078] After receiving the information source collection decision, a targeted crawler task is launched to access the designated content section of Security Technology Forum B and obtain page data, including text posts, accompanying images, and technical demonstration videos, by simulating browser requests.

[0079] The acquired information is parsed: HTML text content is extracted into plain text strings; images are named and saved as JPG format according to source-date-serial number; video files retain their original MP4 format and metadata such as duration and resolution is extracted; at the same time, redundant information such as invalid links and pop-up ads are filtered to ensure data integrity.

[0080] After parsing, the data is integrated according to the structure of text + image + video + metadata to form the raw data of the target information source, providing complete data support for subsequent reward value calculation.

[0081] Figure 3 This is a specific example diagram illustrating the calculation of the reward value of the target information source in an embodiment of the present invention, as shown below. Figure 3 As shown, calculating the reward value for the target data source based on its raw data can include:

[0082] Step 301: Obtain the information release time, related keywords, and information dissemination data from the original data of the target information source;

[0083] Step 302: Based on the information release time, determine whether the information is appearing for the first time. If so, assign a timeliness reward value to the information.

[0084] Step 303: Based on information-related keywords, match the preset high-risk information rule base to determine the information severity reward value;

[0085] Step 304: Determine the information influence reward value based on information dissemination data;

[0086] Step 305: The reward values ​​for information timeliness, information severity, and information influence are weighted and summed according to preset weights to obtain the reward value for the target information source.

[0087] In this embodiment, the reward value for the target information source is obtained by weighting and summing the information timeliness reward value, information severity reward value, and information influence reward value according to preset weights. This may include:

[0088] Each of the information timeliness reward value, information severity reward value, and information influence reward value is configured with a corresponding preset weight coefficient, and the sum of each weight coefficient is 1.

[0089] Each reward value is multiplied by its corresponding weight coefficient, and the three results are added together to obtain the reward value of the target information source.

[0090] In a specific embodiment, core evaluation data is extracted:

[0091] Extract the information release time from the raw data of the target information source, such as the post release timestamp, article online time, and information-related keywords, that is, extract them from the text using NLP word segmentation technology, such as "Shield System" and "remote code execution vulnerability", and information dissemination data, including the number of reposts, comments, and views in a short period of time, as well as the growth rate of each indicator.

[0092] Calculate the reward value for information timeliness:

[0093] The extracted information, corresponding to the time of publication, is compared with the existing information in the historical database using a hash value or SimHash algorithm to determine whether the information is appearing for the first time on the entire network.

[0094] If it is the first time it appears, the information timeliness reward value is +10; if the same or highly similar information already exists, the timeliness reward value is 0.

[0095] Calculate the reward value for information severity:

[0096] A pre-defined high-risk information rule base is established, which includes a list of high-risk event keywords, such as "data breach," "system attack," and "high-risk vulnerability," as well as corresponding severity levels. When implementing this system, the CVE scoring system can be used as a reference, dividing the system into four levels: low, medium, high, and extremely high.

[0097] The extracted information and related keywords are matched with the rule base. If a low-level high-risk keyword is matched, a reward value of +5 is given; for medium-level, +10; for high-level, +15; for very high-level, +20; and if no high-risk keyword is matched, the reward value is 0.

[0098] Calculate the information influence reward value:

[0099] The statistical period is 2 hours after the information is released. The growth rate of forwards, comments and views is calculated. The calculation formula can be: growth rate = (data within the statistical period - data at the time of initial release) / data at the time of initial release.

[0100] The influence reward value is set to a range of +1 to +5. For example, +1 is awarded when the growth rate is ≤10%; +2 is awarded when 10% < growth rate ≤30%; +3 is awarded when 30% < growth rate ≤50%; +4 is awarded when 50% < growth rate ≤80%; and +5 is awarded when the growth rate >80%.

[0101] The final reward value is obtained by weighted summation:

[0102] Preset weight coefficients are configured for the reward values ​​of the three dimensions, where the information timeliness weight w1=0.3, the information severity weight w2=0.5, and the information influence weight w3=0.2, and the sum of each weight coefficient is 1.

[0103] The reward value is calculated using the formula: Reward Value = w1 × Timeliness Reward Value + w2 × Severity Reward Value + w3 × Influence Reward Value. For example, if a piece of information is first appearance (+10), extremely high-level high-risk event (+20), and growth rate of 60% (+4), then the final reward value = 0.3 × 10 + 0.5 × 20 + 0.2 × 4 = 3 + 10 + 0.8 = 13.8.

[0104] If no valuable information is collected, the final reward value is set to -0.1, which is deducted as the collection cost.

[0105] In step 105, an information source experience group is constructed; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, original information source data, and information source reward values.

[0106] In a specific embodiment, a source experience group is constructed, comprising a state vector of the public opinion monitoring source database, source collection decisions, raw source data, and source reward values, including:

[0107] Component data organization:

[0108] Extract the state vector of the public opinion monitoring information source database for the current decision-making cycle, which is the matrix generated in step 102 that consists of the comprehensive state features of all information sources;

[0109] Record the complete information source collection decision determined in step 201, including the target information source, such as "Security Technology Forum B" and the collection method, such as "targeted crawling of content containing the keyword 'Shield System' in the past 72 hours";

[0110] Summarize the original source data parsed in step 202, including plain text strings, JPG format images, MP4 format videos, and corresponding metadata, such as publication timestamps, file sizes, and resolutions;

[0111] Extract the final reward value calculated in step 305, such as 13.8 points, -0.1 points, or other quantitative results.

[0112] Experience group structured encapsulation:

[0113] Following a fixed order of "state vector - acquisition decision - raw data - reward value", the above four types of data are encapsulated into a structured source experience group of <state vector S_t, acquisition decision a_t, raw data D_t, reward value R_t>.

[0114] Add a unique timestamp identifier to the experience group to distinguish the experience data from different decision-making cycles and ensure the accuracy of data traceability during subsequent training;

[0115] The encapsulated experience groups are format-validated to ensure that there are no missing data and that the fields match correctly. Then, they are stored in the experience replay pool to provide complete interactive experience data support for policy network training.

[0116] Figure 4 This is a specific example diagram illustrating the training convergence metric of the policy network in an embodiment of the present invention, as shown below. Figure 4 As shown, based on the empirical set of information sources, the training convergence metric for the policy network can include:

[0117] Step 401: Store the source experience groups into a preset experience replay pool, randomly sample a preset number of source experience groups from the experience replay pool, and calculate the predicted benefit value of each sampled source experience group.

[0118] Step 402: Calculate the training convergence index of the policy network based on the predicted revenue values ​​of each sampled source experience group.

[0119] In a specific embodiment, an empirical group is sampled and the predicted return value is calculated:

[0120] The completed source experience group <state vector S_t, acquisition decision a_t, raw data D_t, reward value R_t> is stored in a fixed-size experience replay pool, for example, with a capacity of 10,000 entries. When the pool is full, the oldest experience is removed according to the first-in, first-out principle.

[0121] A preset number of source experience groups are randomly sampled from the experience replay pool as training batches, for example, the batch size is set to 32, to ensure that the sampled data have no temporal correlation.

[0122] For each sampled experience group, extract its state vector S_j and acquisition decision a_j, input them into the current policy network (DQN), and the network outputs the expected cumulative reward corresponding to the state-action, i.e., the predicted reward value Q(S_j, a_j).

[0123] Computational policy network training convergence metric:

[0124] For each sampled empirical group, the target return value is calculated according to the Bellman equation:

[0125] y_j = R_j + γ × max a 'Q_target(S_{j+1},a')

[0126] Where R_j is the reward value in the experience group, γ is the discount factor (set to 0.9), Q_target is the independent target network, S_{j+1} is the next state vector corresponding to the experience group, and max a 'Q_target(S_{j+1},a') represents the Q value of the next-state optimal action output by the target network.

[0127] The mean squared error is used as the training convergence metric, and the calculation formula is as follows:

[0128] Convergence index = (1 / B) × Σ(y_j - Q(S_j, a_j))²,

[0129] Where B is the sampling batch size, and Σ is the summation of the calculation results for 32 sample experience groups.

[0130] This convergence metric quantifies the degree of deviation between the predicted return value and the target return value of the policy network. The smaller the value of the metric, the more accurate the network prediction is and the closer the training is to the convergence state.

[0131] In step 107, the training parameters of the policy network are updated according to the training convergence index of the policy network.

[0132] In a specific embodiment, updating the training parameters of the policy network according to its training convergence metric includes:

[0133] Convergence status judgment: A preset training convergence threshold is set, for example, mean square error ≤ 0.01. The convergence index calculated in step 402 is compared with this threshold. If the convergence index is less than or equal to the threshold, the strategy network training is determined to be converged, and no parameter update is required; if the convergence index is greater than the threshold, the parameter update process is started.

[0134] Target network parameter synchronization: A separate target network (Q_target) is set up in the system, whose network structure is completely consistent with the policy network (DQN). Every fixed number of training iterations, such as 100, the current training parameters of the policy network are synchronized to the target network to ensure that the target network parameters are updated slowly and improve training stability.

[0135] Parameter gradient update: The gradient descent algorithm is adopted, and the convergence index calculated in step 402 is used as the loss function. The gradient of the parameters of each layer of the policy network is calculated through the backpropagation algorithm. The parameter values ​​are adjusted according to the gradient direction. The adjustment step size is controlled by the learning rate of the optimizer to minimize the loss function.

[0136] Parameter update constraints: The updated network parameters are subject to range constraints to prevent the network training from oscillating due to excessively large or small parameter values. At the same time, the changes in the convergence index before and after the parameter update are recorded. If the convergence index does not decrease after the update, the learning rate is appropriately reduced and the parameter update step is repeated.

[0137] Iterative updates until convergence: After the parameters are updated, return to step 101 to restart a new round of decision and training loop, and continue iterating until the training convergence index of the policy network meets the preset threshold, so as to ensure that the policy network can output stable and optimal information source acquisition decisions.

[0138] Verification has shown that the method for optimizing the information source collection strategy of the public opinion monitoring information source database in this embodiment of the invention has the following beneficial effects:

[0139] 1. Adaptable to dynamic network environments: It overcomes the shortcomings of static lag in existing technology evaluation models. Through a deep learning framework, the agent can continuously interact and learn with the public opinion environment. It can perceive the dynamic changes in the value of information sources in real time and autonomously optimize the strategy network parameters without manual re-labeling and offline retraining. It realizes online adaptive adjustment of the collection strategy and always maintains optimal adaptability to the current environment.

[0140] 2. Improve the capture rate of high-value intelligence: Construct a closed-loop feedback mechanism with intelligence value as the core. Through a multi-dimensional reward function based on timeliness, severity, and impact, the actual monitoring effect of the collection action is directly converted into reward signals. This solves the problem of the disconnect between the existing technology evaluation results and the actual monitoring effect, and enables the capture of key early warning information related to production events and cybersecurity events earlier and more accurately.

[0141] 3. Stable and efficient learning process, reducing manual dependence costs: By storing interactive experience in the experience replay pool and randomly sampling for training, the temporal correlation of data is broken. Combined with the asynchronous updates of the target network and the policy network, the stability and learning efficiency of model training are improved. No manual intervention is required for source evaluation and policy adjustment throughout the process, which greatly reduces the cost of manual annotation and model iteration, and significantly improves the automation level and practical effectiveness of the public opinion monitoring system.

[0142] This invention also provides a device for optimizing the source acquisition strategy of a public opinion monitoring source database, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the method for optimizing the source acquisition strategy of a public opinion monitoring source database, the implementation of this device can refer to the implementation of the method for optimizing the source acquisition strategy of a public opinion monitoring source database; repeated details will not be elaborated further.

[0143] Figure 5 This is a structural example diagram of the information source acquisition strategy optimization device for the public opinion monitoring information source database in an embodiment of the present invention, as shown below. Figure 5As shown, the information source collection strategy optimization device for this public opinion monitoring information source database includes:

[0144] Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold:

[0145] The information acquisition module 501 is used to acquire source characteristic information from the public opinion monitoring source database. The source characteristic information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content.

[0146] The state vector determination module 502 is used to determine the state vector of the public opinion monitoring source database based on the source feature information.

[0147] The acquisition module 503 is used to input the state vector of the public opinion monitoring source database into the strategy network to obtain the source acquisition decision, and execute the source acquisition decision to obtain the original data of the target source; the source acquisition decision includes the target source and the acquisition method; the strategy network is trained on a deep learning network based on the historical state vector of the public opinion monitoring source database.

[0148] The reward value calculation module 504 is used to calculate the reward value of the target information source based on the raw data of the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision.

[0149] The information source experience group construction module 505 is used to construct information source experience groups; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, information source raw data, and information source reward values;

[0150] The training convergence index calculation module 506 is used to calculate the training convergence index of the policy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network.

[0151] The training parameter update module 507 is used to update the training parameters of the policy network according to the training convergence index of the policy network.

[0152] In one embodiment, the state vector determination module 502 is specifically used for:

[0153] Based on the source category, the source category state vector is obtained through one-hot coding;

[0154] Based on the source content, the source content state vector is obtained through a natural language processing model;

[0155] Based on the historical average value of the information source, the state vector of the historical average value of the information source is obtained through normalization.

[0156] Based on the source acquisition success rate, the source acquisition success rate state vector is obtained through normalization processing;

[0157] Based on the source acquisition time, the source acquisition time state vector is obtained through normalization processing.

[0158] In one embodiment, the acquisition module 503 is specifically used for:

[0159] The state vector of the public opinion monitoring source database is input into the policy network, and a greedy algorithm is used to determine the source collection decision:

[0160] The system executes information source acquisition decisions, obtains text, image, or video information from the target information source, and parses the format to obtain the raw data of the target information source.

[0161] In one embodiment, the reward value calculation module 504 is specifically used for:

[0162] Obtain the information release time, related keywords, and information dissemination data from the original data of the target information source;

[0163] Based on the time of information release, determine whether the information is appearing for the first time; if so, assign a timeliness reward value to the information.

[0164] Based on information-related keywords, a pre-defined high-risk information rule base is matched to determine the information severity reward value;

[0165] Determine the reward value for information influence based on information dissemination data;

[0166] The reward value for the target information source is obtained by weighting and summing the reward values ​​for information timeliness, information severity, and information influence according to preset weights.

[0167] In one embodiment, the reward value calculation module 504 is specifically used for:

[0168] Each of the information timeliness reward value, information severity reward value, and information influence reward value is configured with a corresponding preset weight coefficient, and the sum of each weight coefficient is 1.

[0169] Each reward value is multiplied by its corresponding weight coefficient, and the three results are added together to obtain the reward value of the target information source.

[0170] In one embodiment, the training convergence metric calculation module 506 is specifically used for:

[0171] The source experience groups are stored in a preset experience replay pool. A preset number of source experience groups are randomly sampled from the experience replay pool, and the predicted benefit value of each sampled source experience group is calculated.

[0172] The training convergence index of the policy network is calculated based on the predicted revenue values ​​of each sampled source experience group.

[0173] Based on the aforementioned inventive concept, such as Figure 6 As shown, the present invention also proposes a computer device 600, including a memory 610, a processor 620, and a computer program 630 stored in the memory 610 and executable on the processor 620. When the processor 620 executes the computer program 630, it implements the aforementioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0174] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0175] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned method for optimizing the information source collection strategy of the public opinion monitoring information source database.

[0176] In this embodiment of the invention, source feature information from the public opinion monitoring source database is obtained. This source feature information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content. Based on the source feature information, a state vector of the public opinion monitoring source database is determined. The state vector of the public opinion monitoring source database is input into a policy network to obtain a source collection decision. Executing the source collection decision yields the raw data of the target source. The source collection decision includes the target source and the collection method. The policy network is trained on a deep learning network based on the historical state vector of the public opinion monitoring source database. Based on the raw data of the target source, a reward value for the target source is calculated. The reward value is used... This invention evaluates the value of public opinion information corresponding to information source collection decisions; constructs an information source experience group; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, original information source data, and information source reward values; calculates the training convergence index of the strategy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the strategy network; and updates the training parameters of the strategy network based on the training convergence index of the strategy network. This embodiment of the invention, through the combination of deep learning and public opinion collection optimization models, determines an intelligent collection scheme that adapts to the dynamics of information sources, balances collection efficiency and value, and improves decision-making accuracy, providing technical support for the coordinated improvement of public opinion monitoring information source management and collection efficiency.

[0177] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0179] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0180] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0181] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for optimizing the source collection strategy of a public opinion monitoring source database, characterized in that, include: Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold: Obtain source characteristic information from the public opinion monitoring source database, including the historical average value of the source, the time of source collection, the source category, the success rate of source collection, and the content of the source; Based on the source characteristic information, determine the state vector of the public opinion monitoring source database; The state vector of the public opinion monitoring source database is input into the policy network to obtain the source collection decision. The source collection decision is executed to obtain the raw data of the target source. The source collection decision includes the target source and the collection method. The policy network is trained on a deep learning network based on the historical state vector of the public opinion monitoring source database. Based on the raw data collected from the target information source, a reward value is calculated for the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision. Construct a source experience group; the source experience group includes the state vector of the public opinion monitoring source database, source collection decisions, source raw data, and source reward value; Based on the information source experience set, the training convergence index of the policy network is calculated; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network. Update the training parameters of the policy network based on the training convergence metric of the policy network.

2. The method as described in claim 1, characterized in that, Based on the source characteristic information, determine the state vector of the public opinion monitoring source database, including: Based on the source category, the source category state vector is obtained through one-hot coding; Based on the source content, the source content state vector is obtained through a natural language processing model; Based on the historical average value of the information source, the state vector of the historical average value of the information source is obtained through normalization. Based on the source acquisition success rate, the source acquisition success rate state vector is obtained through normalization processing; Based on the source acquisition time, the source acquisition time state vector is obtained through normalization processing.

3. The method as described in claim 1, characterized in that, The state vector of the public opinion monitoring source database is input into the policy network to obtain the source collection decision. Executing the source collection decision yields the raw data of the target source, including: The state vector of the public opinion monitoring source database is input into the policy network, and a greedy algorithm is used to determine the source collection decision: The system executes information source acquisition decisions, obtains text, image, or video information from the target information source, and parses the format to obtain the raw data of the target information source.

4. The method as described in claim 1, characterized in that, Based on the raw data from the target data source, calculate the reward value for the target data source, including: Obtain the information release time, related keywords, and information dissemination data from the original data of the target information source; Based on the time of information release, determine whether the information is appearing for the first time; if so, assign a timeliness reward value to the information. Based on information-related keywords, a pre-defined high-risk information rule base is matched to determine the information severity reward value; Determine the reward value for information influence based on information dissemination data; The reward value for the target information source is obtained by weighting and summing the reward values ​​for information timeliness, information severity, and information influence according to preset weights.

5. The method as described in claim 4, characterized in that, The reward value for the target information source is obtained by weighting and summing the reward values ​​for information timeliness, information severity, and information impact according to preset weights, including: Each of the information timeliness reward value, information severity reward value, and information influence reward value is configured with a corresponding preset weight coefficient, and the sum of each weight coefficient is 1. Each reward value is multiplied by its corresponding weight coefficient, and the three results are added together to obtain the reward value of the target information source.

6. The method as described in claim 1, characterized in that, Based on the empirical set of information sources, the training convergence metrics for the policy network are calculated, including: The source experience groups are stored in a preset experience replay pool. A preset number of source experience groups are randomly sampled from the experience replay pool, and the predicted benefit value of each sampled source experience group is calculated. The training convergence index of the policy network is calculated based on the predicted revenue values ​​of each sampled source experience group.

7. A device for optimizing the source acquisition strategy of a public opinion monitoring source database, characterized in that, include: Repeat the following steps until the training convergence metric of the policy network reaches the preset threshold: The information acquisition module is used to acquire source characteristic information from the public opinion monitoring source database. The source characteristic information includes the historical average value of the source, the source collection time, the source category, the source collection success rate, and the source content. The state vector determination module is used to determine the state vector of the public opinion monitoring source database based on the source feature information. The data acquisition module is used to input the state vectors of the public opinion monitoring source database into the strategy network to obtain source acquisition decisions, and execute the source acquisition decisions to obtain the raw data of the target source; the source acquisition decisions include the target source and the acquisition method; the strategy network is trained on a deep learning network based on the historical state vectors of the public opinion monitoring source database. The reward value calculation module is used to calculate the reward value of the target information source based on the raw data collected from the target information source; the reward value is used to evaluate the value of public opinion information corresponding to the information source collection decision. The information source experience group construction module is used to construct information source experience groups; the information source experience group includes the state vector of the public opinion monitoring information source database, information source collection decisions, information source raw data, and information source reward value; The training convergence index calculation module is used to calculate the training convergence index of the policy network based on the information source experience group; the training convergence index is used to quantify the degree of deviation between the predicted output and the target output of the policy network. The training parameter update module is used to update the training parameters of the policy network based on the training convergence metric of the policy network.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1-6.

Images